{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 为什么Bagging能改进模型性能？（10分）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;首先bagging作为集成算法的一种相比之前只采用一颗树的模型有了明显的性能的提高，bagging的集成方法是分别训练多个弱的分类学习器，然后取平均作为最终结果，\n",
    "典型的一个代表就是随机森林。<br>\n",
    "&emsp;&emsp;对于以前的单个决策树而言，由于偏差低（偏差是指预测值与真实值之间的差异，越小越好），方差高（方差是指多个弱学习器训练出来结果的偏差程度，当然越小越好。高偏差，低方差表现为欠拟合，低偏差高方差表现为过拟合），很容易出现过拟合。而集成算法bagging的基本思想是并行的训练一堆的弱分类学习器，最后将结果平均化，正好降低了模型的方差，从而提高了模型的性能。<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 随机森林中随机体现在哪些方面？（10分）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "随机森林中的“随机”体现在两个方面：对数据采样的随机性，即有放回的选取一部分数据来训练一个弱学习器；特征选择的随机性，即对每个数据的特征进行有放回的选取一部分。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 随机森林和GBDT的基学习器都是决策树。请问这两种模型中的决策树有什么不同。（10分）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;这两个算法的构造决策树的方法上是不同的，它先对原始数据进行采样，然后并行的去训练每一个弱学习器，最后的结果是对所有的若学习器训练的结果取平均。每个弱学习器训练的过程之间是独立的，所以他能并行来做。<br>\n",
    "&emsp;&emsp;而GBDT构造决策树是串行构造的，每一颗树都是在前一棵树的基础上建立起来的，后一个学习器能和前一个若学习器形成互补，也就是说后一个学习器能帮助前一个弱学习器，所以他是按顺序学习的，它的后一个学习器相比前一个学习器有了明显的提升，最后组合所有的弱学习器形成一个强学习器.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 请简述LightGBM的训练速度为什么比XGBoost快。（30分）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;在进行分割点选择的时候，XGBoost是需要遍历所有特征以及特征里面的所有阈值，并且还需要对特征先进行排序。这都需要消耗大量的时间。而LightGBM在选择分割点时，采用的是直方图算法。将直方图中梯度变化的点作为候选的阈值分割点，该点的数量是仅仅有限的几个，而且还不需要对特征值进行排序，很大程度缩短了计算的时间。<br>\n",
    "&emsp;&emsp;LightGBM中采用了Goss算法，对样本进行下采样减少了数据的数目，另外还通过EFB互斥特征捆绑的方法，将高维的稀疏特征空间中，将一些互斥的稀疏特征捆绑在一起，减少了建立直方图的原始数据的特征数目，这样建立直方图的速度也就更快了。<br>\n",
    "&emsp;&emsp;LightGBM中还通过了直方图做差的方法（对于一个父节点两个子节点的局部树来说，知道父节点和其中一个子节点的直方图，然后另一个子节点的直方图就可以通过父节点的直方图减去其中一个子节点的直方图得到）来计算叶子节点的直方图，节省了其中部分叶子建立直方图的时间，这样整体的计算速度又得到了提升。<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 采用tfidf特征，采用LightGBM（gbdt）完成Otto商品分类，尽可能将超参数调到最优，并用该模型对测试数据进行测试，提交Kaggle网站，提交排名。（20分）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基础工作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入必要的模块库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T01:27:33.530627Z",
     "start_time": "2020-06-25T01:27:29.398577Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "import matplotlib as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T02:00:53.999407Z",
     "start_time": "2020-06-25T02:00:48.409328Z"
    }
   },
   "outputs": [],
   "source": [
    "train1 = pd.read_csv(\"Otto_FE_train_org.csv\")\n",
    "train2 = pd.read_csv(\"Otto_FE_train_tfidf.csv\")\n",
    "train2 = train2.drop([\"id\", \"target\"], axis = 1)\n",
    "train = pd.concat([train1, train2], axis = 1, ignore_index = False)\n",
    "del train1\n",
    "del train2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T02:01:24.195716Z",
     "start_time": "2020-06-25T02:01:24.172747Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1</th>\n",
       "      <th>feat_2</th>\n",
       "      <th>feat_3</th>\n",
       "      <th>feat_4</th>\n",
       "      <th>feat_5</th>\n",
       "      <th>feat_6</th>\n",
       "      <th>feat_7</th>\n",
       "      <th>feat_8</th>\n",
       "      <th>feat_9</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.075886</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.013158</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.013158</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.315789</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.131579</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.460983</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008244</td>\n",
       "      <td>0.022456</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.124622</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.145988</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 188 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id    feat_1  feat_2  feat_3    feat_4    feat_5  feat_6    feat_7  \\\n",
       "0   1  0.016393     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "1   2  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "2   3  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "3   4  0.016393     0.0     0.0  0.014286  0.315789     0.1  0.131579   \n",
       "4   5  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "\n",
       "     feat_8  feat_9      ...        feat_84_tfidf  feat_85_tfidf  \\\n",
       "0  0.000000     0.0      ...             0.000000       0.075886   \n",
       "1  0.013158     0.0      ...             0.000000       0.000000   \n",
       "2  0.013158     0.0      ...             0.000000       0.000000   \n",
       "3  0.000000     0.0      ...             0.460983       0.000000   \n",
       "4  0.000000     0.0      ...             0.000000       0.124622   \n",
       "\n",
       "   feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  feat_90_tfidf  \\\n",
       "0       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "1       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "2       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "3       0.008244       0.022456            0.0            0.0       0.000000   \n",
       "4       0.000000       0.000000            0.0            0.0       0.145988   \n",
       "\n",
       "   feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0            0.0            0.0            0.0  \n",
       "1            0.0            0.0            0.0  \n",
       "2            0.0            0.0            0.0  \n",
       "3            0.0            0.0            0.0  \n",
       "4            0.0            0.0            0.0  \n",
       "\n",
       "[5 rows x 188 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T02:10:48.261979Z",
     "start_time": "2020-06-25T02:10:48.071252Z"
    }
   },
   "outputs": [],
   "source": [
    "y_train = train['target']\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s) - 1)#改变下标\n",
    "\n",
    "x_train = train.drop([\"id\", \"target\"], axis = 1)\n",
    "feat_names = x_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T02:11:53.833157Z",
     "start_time": "2020-06-25T02:11:53.579810Z"
    }
   },
   "outputs": [],
   "source": [
    "#数据稀疏化\n",
    "from scipy.sparse import  csr_matrix\n",
    "x_train = csr_matrix(x_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 超参数调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T02:15:43.980613Z",
     "start_time": "2020-06-25T02:15:43.977622Z"
    }
   },
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 10000#最大迭代次数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T03:14:02.101742Z",
     "start_time": "2020-06-25T03:14:02.097732Z"
    }
   },
   "outputs": [],
   "source": [
    "#交叉验证分组\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "kfold = StratifiedKFold(n_splits = 3, shuffle = True, random_state = 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 树的数目n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T03:14:05.314344Z",
     "start_time": "2020-06-25T03:14:05.309358Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_n_estimators(params, x_train, y_train, early_stopping_rounds = 10):\n",
    "    #设置参数\n",
    "    lgbm_params = params.copy()\n",
    "    lgbm_params['num_class'] = 9\n",
    "    \n",
    "    #构造训练数据\n",
    "    lgbmtrain = lgbm.Dataset(x_train, y_train)\n",
    "    \n",
    "    #将参数和数据传到 LightGBM的cv函数中\n",
    "    cv_result = lgbm.cv(lgbm_params, #参数,见下面设置\n",
    "                        lgbmtrain, #训练数据\n",
    "                        num_boost_round = MAX_ROUNDS, #弱分类器的数目\n",
    "                        nfold = 3, #交叉验证的折数\n",
    "                        metrics = 'multi_logloss', #评价指标\n",
    "                        early_stopping_rounds = early_stopping_rounds, #模型的性能为提升的次数超过过早停止设置的数值，则停止训练\n",
    "                        seed = 3)\n",
    "    \n",
    "    print('best n_estimators:', len(cv_result['multi_logloss-mean']))\n",
    "    print('best cv score:', cv_result['multi_logloss-mean'][-1])\n",
    "    \n",
    "    return len(cv_result['multi_logloss-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T03:15:55.756930Z",
     "start_time": "2020-06-25T03:14:08.183764Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 380\n",
      "best cv score: 0.477884355807\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "         'objective': 'multiclass',#多类分类问题\n",
    "         'n_jobs': 4,\n",
    "         'learning_rate': 0.1,\n",
    "         'num_leaves': 60,#叶子节点数目\n",
    "         'max_depth': 6,\n",
    "         'max_bin': 127,#特征取值范围\n",
    "         'subsample': 0.7,#列采样比例\n",
    "         'bagging_freq': 1,# bagging的频率\n",
    "         'colsample_bytree': 1,\n",
    "         }\n",
    "n_estimators_1 = get_n_estimators(params, x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 树的最大叶子节点数目num_leaves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T04:24:26.314157Z",
     "start_time": "2020-06-25T04:14:49.950288Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  6.6min remaining:  3.3min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  9.6min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(bagging_freg=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=0.7, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=20,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=380,\n",
       "        n_jobs=4, num_class=9, num_leaves=31, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.7, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'num_leaves': range(50, 90, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score=True,\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "         'objective': 'multiclass',\n",
    "         'num_class': 9,\n",
    "         'n_jobs': 4,\n",
    "         'learning_rate': 0.1,\n",
    "         'n_estimators': n_estimators_1,#上面调出的最优值380\n",
    "         'max_depth': 7,\n",
    "         'max_bin': 127,\n",
    "         'subsample': 0.7,\n",
    "         'bagging_freg': 1,\n",
    "         'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent = False, **params)#生成一个学习器\n",
    "num_leaves_s = range(50, 90, 10)#调优参数的范围\n",
    "tuned_parameters = dict(num_leaves = num_leaves_s)\n",
    "grid_search = GridSearchCV(lg, \n",
    "                           n_jobs = 4, \n",
    "                           param_grid = tuned_parameters, #要调优参数的范围\n",
    "                           cv = kfold, \n",
    "                           scoring = \"neg_log_loss\",#评价指标\n",
    "                           verbose = 5,\n",
    "                           refit = False,\n",
    "                           return_train_score = True)\n",
    "grid_search.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T04:24:46.234103Z",
     "start_time": "2020-06-25T04:24:46.231075Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.481855355625\n",
      "{'num_leaves': 60}\n"
     ]
    }
   ],
   "source": [
    "#查看当前的最佳分数和最佳参数\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T04:39:46.501688Z",
     "start_time": "2020-06-25T04:39:46.380978Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Log loss')"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAELCAYAAADkyZC4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl4VeXV9/HvyjwSxiCEQAIkQJhEIqgIIgIPgmAVbNVqbfWtQ8U6oEWtVdRaFVFrq3Vo1Wofq1ZAy2CZRVARCVMYAiRACEMgTBkggUzr/eMcfCINJEB29jnJ+lxXruTs7OG3PRxX7nvvfd+iqhhjjDF1LcDtAMYYYxomKzDGGGMcYQXGGGOMI6zAGGOMcYQVGGOMMY6wAmOMMcYRVmCMMcY4wgqMMcYYR1iBMcYY44ggtwO4qWXLlpqQkOB2DGOM8SsrV648oKqtalqvUReYhIQE0tLS3I5hjDF+RUR21GY96yIzxhjjCCswxhhjHGEFxhhjjCOswBhjjHGEFRhjjDGOsAJjjDHGEVZgjDHGOMIKzFnYtLeQj1fkuB3DGGN8mhWYs/D+sh089tl6dh0udjuKMcb4LCswZ+GeIZ0REV5dlOV2FGOM8VlWYM5Cm5hwbuzXnk9W7iL7wFG34xhjjE9ytMCIyAgR2SwiWSLy8GnWGyciKiKp3tfBIvKeiKwTkQwRecS7PF5EvvAu2yAi91azrwe9+2rp3JnBry7vRHCg8KeFmU4exhhj/JZjBUZEAoHXgCuBFOAGEUmpZr1o4NfA8iqLrwNCVbUn0Be4Q0QSgHJggqp2Ay4C7q66TxGJB4YBjl+Bj40O45aLE/hszW6y8oqcPpwxxvgdJ1sw/YAsVd2mqqXAR8DV1az3NDAZOFZlmQKRIhIEhAOlQKGq5qrqKgBVLQIygLgq270M/Ma7vePuuKwT4cGBvLzAWjHGGHMyJwtMHLCzyutd/LAYICJ9gHhVnXXStlOBo0AuntbIFFU9dNK2CUAfvC0fERkD7FbVtXV3CqfXPDKEWy9NZHZ6Lhm5hfV1WGOM8QtOFhipZtn3LQsRCcDT4phQzXr9gAqgLZAITBCRjlW2jQKmAfepaqGIRAC/BR6vMZTI7SKSJiJp+/fvP5Pzqdb/u7Qj0WFBvDR/yznvyxhjGhInC8wuIL7K63bAniqvo4EewGIRycZzTWWG90L/jcAcVS1T1Tzga+D7GwDwFJcPVHW6d1+d8BSitd59tQNWich5J4dS1bdUNVVVU1u1qnFCthrFRATzy4Edmb9xH+m78s95f8YY01A4WWBWAEkikigiIcD1wIwTv1TVAlVtqaoJqpoAfAuMUdU0PN1iQ8QjEk/x2SQiArwNZKjqS1X2tU5VY6vsaxdwgarudfD8vveLAQk0iwi2VowxxlThWIFR1XJgPDAXz8X4f6nqBhF5ynu95HReA6KA9XgK1buqmg4MAG7GU3zWeL9GOnUOtRUdFswdl3Vi8eb9rNxxqOYNjDGmERDVernhyielpqZqWlpaneyruLScQZMXk9w6in/+8qI62acxxvgiEVmpqqk1rWdP8teRiJAgfjW4E99sPcg3Ww+4HccYY1xnBaYO3di/Pec1CeOleVtozC1DY4wBKzB1Kiw4kPFDOpO24zBLMq0VY4xp3KzA1LEfp8bTrlk4L87bbK0YY0yjZgWmjoUEBfDrK5JI31XAgow8t+MYY4xrrMA44No+cSS2jOSl+VuorLRWjDGmcbIC44CgwADuvSKJjNxC/rO+Xp71NMYYn2MFxiGje7clKTaKlxdsocJaMcaYRsgKjEMCA4T7hyWTlXeEGWt3ux3HGGPqnRUYB43ofh4pbZrwyoJMyisq3Y5jjDH1ygqMgwIChAeGJZN9sJjpq6wVYxqfbfuPsCrnsNsxjEuswDjsim6x9I5vyisLMyktt1aMaTxKSiu4+e3vuOGtb9m2/4jbcYwLrMA4TESYMCyZ3fklfJy2s+YNjGkg3lyyld35JQQGCL+Zmm637DdCVmDqwcCkllyY0IxXF2VyrKzC7TjGOG53fglvfLmVUb3a8PTVPUjbcZj3lmW7HcvUMysw9UBEmDC8C/sKj/PB8hy34xjjuGc/z0AVHh3ZjWsviGNwl1ZMnrOZHQePuh3N1CMrMPXkoo4tGNC5Ba8vzqK4tNztOMY4Zvm2g8xKz+XOyzoR1zQcEeHZa3sSFCBMnGZdZY2JFZh69MCwLhw4Usp73+xwO4oxjqioVJ6cuZG2MWHceVmn75e3iQnnt6O68e22Q/zzO2vFNxZWYOpR3w7NuLxLK95cspWiY2VuxzGmzn28Yicbcwt5dFQ3wkMCf/C7n1wYz6WdW/Ls5xnsOlzsUkJTn6zA1LMHhnUhv7iMd77KdjuKMXWqoLiMKfM20y+xOaN6tvmv35/oKlPgkenrbDqLRsAKTD3r2S6G4Smt+dtX2ygotlaMaTheWZjJ4eJSnhidgohUu0588wgeubIrSzMP8EnarnpOaOqbFRgXPDA8mSPHy/nr0m1uRzGmTmTlFfH+smyuv7A93dvGnHbdn/bvQP/E5jw9eyN7C47VT0DjCiswLuh6XhNG9WzDO19v5+CR427HMeacqHou7IeHBPLg8OQa1w8IEJ4f24uyikoe/dS6yhoyKzAuuW9oMsfKKnhzibVijH9bmJHH0swD3D80mRZRobXaJqFlJA/9T1cWbcrj09U2Tl9DZQXGJZ1jo/hRnzje+yabvELrJjD+6Xh5BU/P3kjn2ChuvrjDGW3780sS6NuhGU/O3EhekX0GGiIrMC6694okyiuVvyze6nYUY87Ku19ns+NgMY9flUJw4Jn97yTQ21VWUlbB7z5bb11lDZAVGBd1aBHJj1Pb8c/lOezJL3E7jjFnJK/wGH9emMnQbq0ZlNzqrPbROTaKB4YlM3fDPmal59ZxQuM2RwuMiIwQkc0ikiUiD59mvXEioiKS6n0dLCLvicg6EckQkUe8y+NF5Avvsg0icm+VfTwtIukiskZE5olIWyfPra6MH5IEwJ8XZbmcxJgz8/yczZRVKI+N6nZO+/l/lybSu10MT8zYYDe9NDCOFRgRCQReA64EUoAbRCSlmvWigV8Dy6ssvg4IVdWeQF/gDhFJAMqBCaraDbgIuLvKPl9Q1V6qej4wC3jckROrY3FNw7mhXzyfpO0k56A93Wz8w5qd+UxbtYtbL00koWXkOe0rKDCAF67rTdGxMp6YsaGOEhpf4GQLph+QparbVLUU+Ai4upr1ngYmA1Wv8ikQKSJBQDhQChSqaq6qrgJQ1SIgA4jzvi6ssn2kdx9+4e7LOxMYILyyMNPtKMbUqLJSmTRjA62iQxk/pHOd7DO5dTS/HpLErPRc5qzfWyf7NO5zssDEAVVn2NrlXfY9EekDxKvqrJO2nQocBXKBHGCKqh46adsEoA9VWj4i8oyI7AR+ip+0YABim4Rx80Ud+HT1LrbazH/Gx326ejdrduYzcURXokKD6my/dw7uRPe2TXjss/UcPlpaZ/s17nGywFQ3VsT3rQoRCQBeBiZUs14/oAJoCyQCE0SkY5Vto4BpwH1VWy6q+ltVjQc+AMZXG0rkdhFJE5G0/fv3n/lZOeTOwZ0ICw7klQXWijG+68jxcp6fs4ne8U25tk9czRucgeDAACaP60V+cSlPz9pYp/s27nCywOwC4qu8bgfsqfI6GugBLBaRbDzXVGZ4L/TfCMxR1TJVzQO+Br6/AQBPcflAVaef4tj/BMZW9wtVfUtVU1U1tVWrs7vzxQkto0L5+SUJzEzfw+a9RW7HMaZar32RRV7RcSaNTiEgoPrxxs5F97Yx/GpwJ6av3s2iTfvqfP+mfjlZYFYASSKSKCIhwPXAjBO/VNUCVW2pqgmqmgB8C4xR1TQ83WJDxCMST/HZJJ4R9N4GMlT1paoHE5GkKi/HAJscPDdH3D6oI1EhQbw8f4vbUYz5L9kHjvL20u1ce0Ecfdo3c+w444ck0aV1NI9MX0dBiQ0I688cKzCqWo6nm2ounovx/1LVDSLylIiMqWHz14AoYD2eQvWuqqYDA4Cb8RSfNd6vkd5tnhOR9SKSDgwH7q1ux76saUQItw1MZM6GvazfXeB2HGN+4JnPMwgOFB4e0dXR44QEBfDCdb04cKSUP8zOcPRYxlnSmJ+eTU1N1bS0NLdj/EDhsTIGPv8FfTs0452fX+h2HGMAWJq5n5vf/o6JI7py1+BONW9QB56fs4nXF2/l/Vv7nfWDnMYZIrJSVVNrWs+e5PcxTcKCueOyjizalMeqnMNuxzGGsopKnpy5kQ4tIrj10oR6O+69VyTRqVUkj0xfx5Hj5fV2XFN3rMD4oFsuTqBFZAgvzbNrMcZ9//vtDrLyjvDYqBRCgwJr3qCOhAUHMnlcb/YUlPDs59ZV5o+swPigyNAg7hrcia+yDrB820G345hG7OCR47w8fwsDk1oytFtsvR+/b4dm3DYgkQ+W5/DN1gP1fnxzbqzA+KibLupAbHQoL87bYqPMGte8OH8LR0srePyqU0+D7LQJw7uQ0CKCh6eto7jUusr8iRUYHxUWHMj4IZ35LvsQX2XZX26m/m3YU8CH3+Xws4s7kNQ62rUc4SGBPD+2FzmHipk8Z7NrOcyZswLjw35yYTxxTcOtFWPq3YlpkJtFhHDfFTVPg+y0/h1bcMvFHXhvWTYrsg/VuL7xDVZgfFhoUCD3DOnMmp35LNqU53Yc04jMXpfLd9sPMWF4MjERwW7HAeA3I7oS1zSciVPTOVZW4XYcUwtWYHzc2L7taN88gpfmWyvG1I+S0gqe/XwTKW2acP2F7d2O873I0CCeH9uLbQeO8pKNduEXrMD4uODAAO4bmsSGPYXM3WDDmBvnvblkK7vzS3hidAqBDow3di4GdG7JDf3a87el21htz4n5PCswfuDq8+Po1CqSl+ZvoaLSWjHGObvzS3jjy62M6tWG/h1buB2nWo+O7ErrJmE8NDWd4+XWVebLrMD4gcAA4f5hyWzZd4RZ6Xtq3sCYs/Ts5xmowqMjz20aZCdFhwXz7LU9yco7wp9skj6fZgXGT4zs0Yau50XzxwWZlFdUuh3HNEDLtx1kVnoud17Wibim4W7HOa3BXWIZ17cdb3y5jXW7bGBYX2UFxk8EeFsx2w8c5dPVu92OYxqYikrPbcltY8K487L6GczyXP1uVAotIkN4aOpaSsvtjy5fZAXGjwxPaU3PuBj+tCjTPlCmTn28Yicbcwt5dFQ3wkPqb7yxcxETEcwz1/Rk094i/rI4y+04phpWYPyIiPDA8GR2Hirhk5U73Y5jGoiC4jKmzNtMv8TmjOrZxu04Z2RYSmuuPr8try7KIiO3sOYNTL2yAuNnBie3om+HZry6KMseNjN14pWFmeQXl/LEaPfGGzsXk0Z3p2lEMA9NXUuZXZ/0KVZg/IyIMGFYMrkFx/jwuxy34xg/l5VXxPvLsrm+X3u6t41xO85ZaRYZwtNX92D97kLeWrLN7TimCiswfuiSzi25qGNzXvtiKyWl1ooxZ+fEeGPhIYFMGOb+eGPn4sqebRjVsw2vLMgkc1+R23GMlxUYPzVheBcOHDnOP77NdjuK8VMLM/JYmnmA+4cm0yIq1O045+zJq7sTGRrIQ1PT7YFkH2EFxk9dmNCcQcmteH3xVptO1pyx4+UVPD17I51jo7j54g5ux6kTLaNCmTSmO2t25vPOV9vdjmOwAuPXJgxL5nBxGX//2j5M5sy881U2Ow4W8/hVKQQHNpz/DYzp3ZZhKa2ZMm8z2/YfcTtOo9dw/mU1Qr3jmzK0W2veWrKNgpIyt+MYP5FXeIxXF2UytFtrBiW3cjtOnRIRnvlRD0KDApg4LZ1K6ypzlRUYP/fAsGQKj5Xz9lK7e8bUzvNzNlNWoTw2ynfHGzsXsU3CeHx0d1ZkH+a9Zdlux2nUrMD4uZS2TRjVsw3vfJ3NoaOlbscxPm51zmGmrdrFrZcmktAy0u04jhl7QRyDu7Ri8pzN5BwsdjtOo2UFpgG4b2gSR0vLeXPJVrejGB9W6R1vLDY6lPFDOrsdx1EiwrPX9iQoQKyrzEVWYBqApNbRXN27Le99k01e0TG34xgf9enq3azZmc/EEV2JCg1yO47j2sSE8+iobizbdpB/2kPJrnC0wIjICBHZLCJZIvLwadYbJyIqIqne18Ei8p6IrBORDBF5xLs8XkS+8C7bICL3VtnHCyKySUTSReRTEWnq5Ln5mnuHJlNWoby+2Fox5r8dOV7Oc3M20Tu+Kdf0iXM7Tr25/sJ4Lu3ckmc/z2DXYesqq2+OFRgRCQReA64EUoAbRCSlmvWigV8Dy6ssvg4IVdWeQF/gDhFJAMqBCaraDbgIuLvKPucDPVS1F7AFeMSJ8/JViS0jGXtBHB8szyG3oMTtOMbHvPZFFvuLjjNpdAoBPjYNspNOdJUp8Mj0dahaV1l9crIF0w/IUtVtqloKfARcXc16TwOTgap9OwpEikgQEA6UAoWqmquqqwBUtQjIAOK8r+ep6oknDr8F2jlwTj7tniFJqCqvfWFDl5v/k33gKG8v3c7YC9rRp30zt+PUu/jmETx8ZVeWZh7gk7RdbsdpVJwsMHFA1THld3mXfU9E+gDxqjrrpG2nAkeBXCAHmKKqh07aNgHoww9bPifcCvznHLL7pfjmEfzkwng+XrGTnYesO8B4/H52BsGBwsQRXdyO4pqb+negf2Jznp69kb0Fdp2yvjhZYKprh3/fPhWRAOBlYEI16/UDKoC2QCIwQUQ6Vtk2CpgG3KeqP5gEQkR+i6cr7YNqQ4ncLiJpIpK2f//+MzsjPzD+8iREhD8vsrnKDSzZsp8FGfsYPySJ2CZhbsdxTUCA8PzYXpRVVPLbT62rrL44WWB2AfFVXrcD9lR5HQ30ABaLSDaeayozvBf6bwTmqGqZquYBXwPf3wCAp7h8oKrTqx5QRG4BrgJ+qqf4F6Sqb6lqqqqmtmrVsJ5iBjgvJoyb+ndg2qrdbD9w1O04xkVlFZU8NWsjHVpEcOulCW7HcV1Cy0ge+p+uLNyUx2drbNrx+uBkgVkBJIlIooiEANcDM078UlULVLWlqiaoagKe6yZjVDUNT7fYEPGIxFN8NolnNqS3gQxVfanqwURkBDDRu49G3T901+BOhAQG8MqCLW5HMS76x7IdZOUd4bFRKYQG+cc0yE77+SUJXNC+KZNmbLRb+uuBYwXGe8F9PDAXz8X4f6nqBhF5SkTG1LD5a0AUsB5PoXpXVdOBAcDNeIrPGu/XSO82r+JpFc33Ln/DgdPyC62iQ7nlkgT+vXaPzY3RSB08cpyXF2xhYFJLhnaLdTuOzwgMECaP601JWQW/+2y9dZU5TGr6D+x91uRdoAj4G54L6w+r6jzn4zkrNTVV09LS3I7hiMNHSxk4+QsGJbfkLz/t63YcU88e/XQdH6/Yydz7BtI5NtrtOD7njS+38tx/NvHqjX24qldbt+P4HRFZqaqpNa1XmxbMrd4L6cOBVsAvgOfOMZ9xWLPIEG4dkMDn6/ayYU+B23FMPdqwp4APv8vhZxd3sOJyCv/v0kR6t4vh8X9v4OCR427HabBqU2BO3A02Ek9X1Vqqv0PM+JjbBnakSVgQL8+3O8oaixPTIDeLCOG+K/x7GmQnBQUGMHlcb4qOlfHEjA1ux2mwalNgVorIPDwFZq73yftKZ2OZuhATHsztgzqyIGMfa3bmux3H1IPZ63L5bvshJgxPJiYi2O04Pq3LedH8ekgSs9JzmbN+r9txGqTaFJjbgIeBC713ZwXj6SYzfuDnAxJpHhnCS/PtjrKGrqS0gmc/30RKmyZcf2F7t+P4hTsHdyKlTRMe+2w9+cU23UVdq02BuRjYrKr5InIT8Bhgnfp+Iio0iDsv68iSLftZkX2o5g2M33pzyVZ255fwxOgUAhvReGPnIjgwgBeu60V+cSlPzdzodpwGpzYF5nWgWER6A78BdgDvO5rK1KmbL0qgVXQoL87b7HYU45Dd+SW88eVWRvVqQ/+OLdyO41e6t43hV4M7MX31bhZt2ud2nAalNgWm3PtU/NXAK6r6Cp7nTYyfCA8J5O7Bnfh22yG+yTrgdhzjgGc/zwDg0ZENcxpkp40fkkSX1tE8Mn0dBSVlbsdpMGpTYIq887HcDMz2DsNvVw/9zPX92tMmJowp8zbbw2UNzPJtB5mVnsudl3Uirmm423H8UkhQAJPH9WJ/0XH+MDvD7TgNRm0KzE+A43ieh9mLZ0TkFxxNZepcWHAg9wxJYlVOPou3NLxBPhurikpl0syNtI0J445BndyO49d6xzfl9kGd+DhtJ0vsM1Inaiww3qLyARAjIlcBx1TVrsH4oetS2xHfPJyX5m2xVkwD8fGKnWTkFvLoqG6Eh9h4Y+fqvqFJdGoVySPT13HkeHnNG5jTqrHAiMiPge/wzDL5Y2C5iIxzOpipe8GBAfx6SBLrdhcwb6NdzPR3BcVlTJm3mX6JzRnVs43bcRqEsOBAJo/rzZ6CEp77j3WVnavadJH9Fs8zMLeo6s/wzNXyO2djGadc0yeOji0jeXn+FiorrRXjz/64cAv5xaU8MToFz0Djpi707dCMWwck8r/f5vDNVrsp5lzUpsAEeOdkOeFgLbczPigoMIB7hyaxaW8Rs9fluh3HnKXMfUW8v2wH1/drT/e2MW7HaXAeHN6FhBYRPDxtHcWl1lV2tmpTKOaIyFwR+bmI/ByYDXzubCzjpNG92pLcOoqXF2yhvMJG/fE3qspTszYSGRLIhGE23pgTwkMCeX5sL3IOFfPCXHt+7GzV5iL/Q8BbQC+gN/CWqk50OphxTkCA8MCwZLbtP8q/1+ypeQPjUxZk5LE08wD3DU2mRVSo23EarP4dW/Czizvw92+ySbNRMM5Krbq6VHWaqj6gqver6qdOhzLO+5/u59G9bRNeWZhJmbVi/Mbx8gp+P3sjSbFR3HxxB7fjNHgTR3Qlrmk4v5mazrGyCrfj+J1TFhgRKRKRwmq+ikSksD5DmronIkwYnkzOoWKmrdzldhxTS+98lc2Og8U8PjqF4EC7FOq0yNAgnh/bi20HjvKyDRh7xk75L1RVo1W1STVf0arapD5DGmdc3iWW8+Ob8qeFmRwvt7/OfF1e4TFeXZTJ0G6tGZjUyu04jcaAzi25oV97/rp0G6tzDrsdx6/Yn0CN2IlWzJ6CY3y8YqfbcUwNnp+zmbIK5XdX2Xhj9e2RkV1p3SSM30xNtz/GzoAVmEbu0s4t6ZfYnFcXZVkfsw9bnXOYaat2cdvARDq0iHQ7TqPTJCyYZ6/tSWbeEf600GaIrS0rMI2ciDBhWDJ5Rcf53293uB3HVKPSO95YbHQod1/e2e04jdbgLrGM69uON77cxvrdNiVWbViBMfTv2IKBSS15ffFWjtr4Sz7n09W7Wbszn4kjuhIVGuR2nEbtd6NSaBEZwoOfrKW03O6+rEltxiKr7m6ynSLyqYh0rI+QxnkPDEvm4NFS/v5NtttRTBVHjpfz3JxNnB/flGv6xLkdp9GLiQjmmWt6smlvEX9ZnOV2HJ9XmxbMS8BDeIbpbwc8CPwV+Ah4x7lopj71ad+MIV1jeWvJNgqP2YRLvuLVRVnsLzrOE6NTCLBpkH3CsJTWXH1+W15dlEVGrj2xcTq1KTAjVPVNVS1S1UJVfQsYqaofA80czmfq0QPDkikoKePtpdvdjmKA7ANHeeer7Yy9oB192ttHzZdMGt2dphHB/GZqug23dBq1KTCVIvJjEQnwfv24yu9sON4GpEdcDCO6n8c7X20nv7jU7TiN3u9nZxAcKEwc0cXtKOYkzSJDeOrqHqzbXcCbS7a5Hcdn1abA/BTPdMl53q+bgZtEJBwYf7oNRWSEiGwWkSwRefg0640TERWRVO/rYBF5T0TWiUiGd8pmRCReRL7wLtsgIvdW2cd13mWVJ/Zjztz9w5I5UlrOW/ahcdWSLftZkLGP8UOSiG0S5nYcU42RPdswsud5vLIgk8x9RW7H8Um1Gexym6qOVtWW3q/RqpqlqiWq+tWpthORQOA14EogBbhBRFKqWS8a+DWwvMri64BQVe0J9AXuEJEEoByYoKrdgIuAu6vscz1wLbCkxrM2p9TlvGhG92rLu19nc+DIcbfjNEplFZU8NWsjHVpEcOulCW7HMafx5JgeRIYG8tDUdCpsfqX/Upu7yNp57xjLE5F9IjJNRNrVYt/9gCxvgSrFc1PA1dWs9zQwGThWZZkCkSISBIQDpUChquaq6ioAVS0CMvDcfICqZqiqjatdB+4dmsTx8greWLzV7SiN0j+W7SAr7wiPjUohNMimQfZlraJDmTSmO2t25vPOV3bt8mS16SJ7F5gBtMXzP/OZ3mU1iQOqjj+yy7vseyLSB4hX1VknbTsVOArkAjnAFFU9dNK2CUAfftjyMXWgU6sorr2gHf/4dgf7Co/VvIGpMwePHOflBVsYmNSSod1i3Y5jamFM77YM7daaKfM2s23/Ebfj+JTaFJhWqvquqpZ7v/4O1Gakveruqfy+DSkiAcDLwIRq1usHVOApaonAhKrP3IhIFDANuE9Vz+g+QRG5XUTSRCRt//79Z7Jpo3LvFUlUVCqvfWH3+tenF+dvobi0wqZB9iMiwh+u6UFoUAATp6XbVORV1KbAHBCRm0Qk0Pt1E55pk2uyC4iv8rodUHV2q2igB7BYRLLxXFOZ4b1AfyMwR1XLvNM1fw18fwMAnuLygapOr0WOH1DVt1Q1VVVTW7WyEWlPJb55BNelxvPhdznsOlzsdpxGYcOeAj78LoefXdyBzrHRbscxZyC2SRiPj+7OiuzDvL8s2+04PqM2BeZW4MfAXjxdVuOAX9RiuxVAkogkikgIcD2erjYAVLXAe9NAgqomAN8CY1Q1DU+32BDxiMRTfDaJ50+6t4EMVX2p1mdpzso9QzojCK8uslaM01SVJ2dupFlECPcNtWmQ/dHYC+IY3KUVz8/ZTM5B+6MMancXWY6qjlHVVqoaq6o/wnO3Vk3bleO5jXkunovx/1LVDSLylIiMqWHz14AoPHeGrQDeVdV0YACe26SHiMga79dIABG5RkR2ARcDs0Vkbk0Zzem1bRrOjf3b88n8CR2hAAAbA0lEQVTKXew4eNTtOA3a7HW5fLf9EA8O70JMeLDbccxZ8HSV9SQwQKyrzEtUz/w/gojkqGp7B/LUq9TUVE1LS3M7hk/LKzzGoBe+YGTPNrz04/PdjtMglZRWcMWLi2kaEcLMey4l0IaE8WsffpfDI9PX8cw1Pfhp/4Y5rbWIrFTVGp83PNvRlO0T0EjENgnjZxcn8Nnq3WTl2R0yTnhzyVb2FBxj0pjuVlwagOsvjGdA5xY8+/kmdueXuB3HVWdbYKzt14jcMagjYcGB/HGBzUle13bnl/DGl1u5qlcb+iU2dzuOqQMiwnPX9qJSlYenpXM2vUQNxSkLzCmG6S8UkSI8tw+bRqJFVCi3DkhkVnqujR5bx/7weQYAj4y0aZAbkvjmETx8ZVeWZh7gk5W73I7jmlMWGFWNVtUm1XxFq6rNetTI/HJgR6LDgnh5vrVi6srybQeZnZ7LnZd1Iq5puNtxTB27qX8H+iU25+lZG9lb0DgfWLYZLU2txEQE88uBHZm3cR/rdtl0seeqwjsNclzTcO4Y1MntOMYBAQHC5LG9KKuo5LefrmuUXWVWYEyt/WJAAk0jgnlxvg35dq4+WpFDRm4hj4zsSniIjTfWUCW0jOTB4V1YuCmPz9bsdjtOvbMCY2otOiyYOwZ1YvHm/azccajmDUy1CorLmDJ3M/0TmzOqZxu34xiH/WJAIhe0b8qkGRvJK2pcXWVWYMwZueWSDrSMCuHFeXYt5mz9ceEWCkrKeNzGG2sUAgOEyeN6U1JWweOfbWhUXWVWYMwZiQgJ4q7Bnflm60GWba3NkHSmqsx9Rby/bAfX92tP97Yxbscx9aRzbBT3D01mzoa9zF6X63acemMFxpyxn/Zvz3lNwnhp/uZG9dfYuVJVnpq1kciQQB4cbtMgNza/HJhI73YxPP7vDRxsJJP5WYExZywsOJC7h3RmRfZhlmQecDuO31iQkcfSzAPcPyyZ5pEhbscx9SwoMIDJ43pTdKyMSTM3uh2nXliBMWflJ6nxxDUN56V51oqpjePlFfx+9kaSYqO46aKGOT6VqVmX86K5Z0gSM9fuYe6GvW7HcZwVGHNWQoICuPeKJNbuKmBBRp7bcXzeO19ls+NgMY+PTiE40D52jdldgzuR0qYJv/10PfnFpW7HcZT9Szdn7doL4khoEcFL87fY0OSnkVd4jFcXZTIspTUDk2ySu8YuODCAF67rRX5xKU/NathdZVZgzFkLCgzgvqHJZOQWMqcRNPfP1vNzNlNWoTw2ysYbMx7d28Zw1+BOTF+1m0Wb9rkdxzFWYMw5Gd27LUmxUbw0fwsV1or5L6tzDjNt1S5uG5hIhxaRbscxPmT8kM4kt47i0enrKTxW5nYcR1iBMeckMEC4b2gyWXlHmLl2j9txfEqld7yx2OhQ7r68s9txjI8JDQrkhXG9ySs6xjOzMtyO4wgrMOacXdnjPLq1acIfF2yhvKLS7Tg+49PVu1m7M5+JI7oSFWoDkJv/1ju+KbcP6sTHaTtZmrnf7Th1zgqMOWcBAcIDw5LJPljM9FWNb0C/6hw5Xs5zczZxfnxTrukT53Yc48PuG5pEx1aRPDxtHUeOl7sdp05ZgTF1Ymi3WHq3i+GVhZmUllsr5tVFWewvOs4To1MIsGmQzWmEBQfywrhe7Cko4bn/NKyuMiswpk6ICA8M78Lu/BI+TtvpdhxXZR84yjtfbWfsBe3o076Z23GMH+jboTm3Dkjkf7/NaVBj/FmBMXVmUFJLUjs049VFmRwrq3A7jmt+PzuD4EBh4ggbb8zU3oPDu9ChRQQTp6VTXNowusqswJg6IyJMGN6FfYXH+WB5jttxXLFky34WZOxj/JAkYpuEuR3H+JHwkECeH9uLnEPFvDC3YUzqZwXG1KmLO7Xgkk4teH1xVoP5K6y2yioqeWrWRhJaRHDrpQluxzF+6KKOLfjZxR34+zfZpGX7/6R+VmBMnZswPJkDR0p5f9kOt6PUq38s20FW3hEeG5VCaJBNg2zOzsQRXWkbE85vpqb7fVezFRhT5/p2aM7gLq1448utFDXQJ5RPdvDIcV5esIWBSS25olus23GMH4sMDeL5sb3YduAoL8/375ljHS0wIjJCRDaLSJaIPHya9caJiIpIqvd1sIi8JyLrRCRDRB7xLo8XkS+8yzaIyL1V9tFcROaLSKb3u92+46IHhiWTX1zGu19nux2lXrw4fwslpRU8YdMgmzpwaVJLbugXz1+XbmN1zmG345w1xwqMiAQCrwFXAinADSKSUs160cCvgeVVFl8HhKpqT6AvcIeIJADlwARV7QZcBNxdZZ8PAwtVNQlY6H1tXNKrXVOGp7Tmr0u3UVDcsFsxG/YU8OF3Ofzs4gQ6x0a7Hcc0EI+M7EbrJmH8Zmo6x8v9s6vMyRZMPyBLVbepainwEXB1Nes9DUwGjlVZpkCkiAQB4UApUKiquaq6CkBVi4AM4MRj0lcD73l/fg/4UR2fjzlD9w9LpuhYOX9dus3tKI5RVZ6csZFmESHcOzTJ7TimAWkSFswfru1JZt4R/rwwy+04Z8XJAhMHVH3ibhf/VwwAEJE+QLyqzjpp26nAUSAXyAGmqOqhk7ZNAPrwfy2f1qqaC+D9bh3hLuvWpgmjerXh3a+3N9g5yGevy+W77EM8OLwLMeHBbscxDczlXWIZe0E7Xv9yK+t3F7gd54w5WWCq64j+fjx3EQkAXgYmVLNeP6ACaAskAhNEpGOVbaOAacB9qlp4RqFEbheRNBFJ27+/4Q0u52vuH5pESVkFby5peK2YktIK/jA7g5Q2TfjJhfFuxzEN1ONXpdAiMoQHP1nrd8MwOVlgdgFVP3XtgKrjuUcDPYDFIpKN55rKDO+F/huBOapapqp5wNfA9zcA4CkuH6jq9Cr72ycibbzrtAGqncdXVd9S1VRVTW3VymYXdFrn2Gh+dH4c7y/LJq/oWI3r+5M3vtzKnoJjTBrTnUAbb8w4JCYimGeu6cmmvUW8vnir23HOiJMFZgWQJCKJIhICXA/MOPFLVS1Q1ZaqmqCqCcC3wBhVTcPTLTZEPCLxFJ9N4rk9520gQ1VfOul4M4BbvD/fAvzbwXMzZ+DeoUmUVSh/+cK/Phynszu/hDe+3MpVvdrQL7G523FMAzcspTVjerfl1S8y2bT3jDptXOVYgVHVcmA8MBfPxfh/qeoGEXlKRMbUsPlrQBSwHk+heldV04EBwM14is8a79dI7zbPAcNEJBMY5n1tfECHFpFc17cd/1yew578Erfj1Ik/fJ6BiOdOH2Pqw6Qx3YkJD+ahT9L9Zt4lUW2809ympqZqWlqa2zEahV2Hi7l8ymKuS43nD9f0dDvOOfl220Guf+tb7huaxH1Dk92OYxqRz9fl8qsPVvGbEV341WD3ZkkVkZWqmlrTevYkv6kX7ZpFcEO/9vxrxU5yDha7HeesVVQqT87cSFzTcO4Y1MntOKaRGdmzDVf2OI8/zs8kK6/I7Tg1sgJj6s3dl3cmMED406JMt6OctY9W5JCRW8ijI7sRHmLjjZn699TVPYgMDeShqelUVPp2D5QVGFNvWjcJ4+aLOjB91S627T/idpwzVlBcxpS5m+mf2JyRPc9zO45ppFpFhzJpTHdW5+Tzzlfb3Y5zWlZgTL26c3AnwoID+eMC/2vF/HHhFgpKynhidHcbb8y4akzvtgzt1pop8zaz/cBRt+OckhUYU69aRoVyyyUJzEzfw+a9vt+HfELmviLeX7aDG/q1J6VtE7fjmEZORHjmmh6EBgUwcWo6lT7aVWYFxtS7OwZ1JCokyG+GIldVnpq1kciQQCYMt2mQjW9o3SSM312VwnfZh3h/WbbbcaplBcbUu6YRIdx6aSJzNuz1i/GVFmTksTTzAPcPS6Z5ZIjbcYz53ri+7bgsuRXPz9nMzkO+d3emFRjjitsGJhITHuzzrZjj5RX8fvZGkmKjuOmiDm7HMeYHRIRnr+1JYIAwcVo6vvZcoxUY44omYcHcPqgjCzflscqHJ1R656tsdhws5vHRKQQH2sfF+J62TcN5dGQ3vtl6kH9+l+N2nB+wT4xxzc8vSaBFZIjPtmLyCo/x6qJMhqW0ZmCSDYxqfNcN/eIZ0LkFz36+id0+NByTFRjjmsjQIO4a3ImlmQdYvu2g23H+y3NzNlFWoTw2ysYbM75NRHju2l5UqvLI9HU+01VmBca46qaLOhAbHcqL87f4zIcCYHXOYaav2s1tAxPp0CLS7TjG1Ci+eQQTR3RlyZb9fLJyl9txACswxmVhwYHcfXlnvtt+iK+zfKMVU1mpTJq5kdjoUO6+3L0BBY05Uzdf1IF+ic15etZG9ha4P/+SFRjjuuv7xdM2Jowp8zb7RCtm+urdrN2Zz8QRXYkKDXI7jjG1FhAgTB7bi7KKSn77qftdZVZgjOtCgwK554ok1uzM54vN1U5EWm+OHC/n+TmbOD++Kdf0iXM1izFnI6FlJA8O78LCTXn8e82emjdwkBUY4xPG9W1H++YRvDjP3Wsxry7KYn/RcSaN6U6ATYNs/NQvBiRyQfumTJq5wdWpyq3AGJ8QHBjAvVcksWFPIXM37HUlQ/aBo7zz1XbGXtCO8+ObupLBmLoQGCBMHteb4tIKHv9sg2t/tFmBMT7jR33i6Ngqkpfmb3Flnovfz84gOFCYOMLGGzP+r3NsFPcPTWbOhr18vs6dP9qswBifERgg3D80mS37jjArvX77jpds2c+CjH3cc0USsU3C6vXYxjjllwMT6dUuhsf/vZ6DR47X+/GtwBifMqpnG7qeF80rCzIpr6isl2OWVVTy1KyNJLSI4BcDEurlmMbUh6DAAF4Y15vCY2VMmrmx3o9vBcb4lIAA4f5hyWw7cJRPV++ul2P+Y9kOsvKO8NioFEKDbBpk07B0OS+ae4YkMXPtnnq/vmkFxvic4Smt6RkXw58WZVLmcCvm4JHjvLxgC4OSW3FFt1hHj2WMW+4a3IlubZrw2GfryS8urbfjWoExPkdEeGBYMjsPlfBJmrNDXkyZt4WS0goev6qbTYNsGqzgwABeGNeLw0dLeWpW/XWVWYExPmlwl1Zc0L4pf16UybGyCkeOsWFPAR+tyOFnFyfQOTbakWMY4yt6xMVw1+BOTF+1m0Wb9tXLMa3AGJ8kIkwY3oXcgmN85MAcF6rKkzM20iwihHuHJtX5/o3xReOHdCa5dRSPTl9P4bEyx49nBcb4rEs6taB/YnNe/WIrJaV124qZlZ7Ld9mHeHB4F2LCg+t038b4qtCgQF4Y15v8klJWbD/k+PGswBifdaIVc+DIcf7xbXad7bektIJnP8+ge9sm/OTC+DrbrzH+oHd8U76eOIQrurV2/FiOFhgRGSEim0UkS0QePs1640RERSTV+zpYRN4TkXUikiEij1RZ9x0RyROR9Sfto7eILPNuM1NEmjh3Zqa+9EtszsCklrzx5TaOHC+vk32+8eVW9hQc44nR3Qm08cZMI9QiKrRejuNYgRGRQOA14EogBbhBRFKqWS8a+DWwvMri64BQVe0J9AXuEJEE7+/+Doyo5pB/Ax72bvMp8FCdnIhx3YThXTh0tJS/f739nPe163Axb3y5lat6taFfYvM6SGeMORUnWzD9gCxV3aaqpcBHwNXVrPc0MBmoOuSnApEiEgSEA6VAIYCqLgGq6zzsAizx/jwfGFsXJ2Hcd358U4Z2i+WtJdsoKDm3C5PP/mcTIvDoSJsG2RinOVlg4oCdVV7v8i77noj0AeJVddZJ204FjgK5QA4wRVVruiK1Hhjj/fk6oNrOdRG5XUTSRCRt//79tToR4777hyVTeKyct786+1bMt9sOMjs9l7su60zbpuF1mM4YUx0nC0x1ndvfD5ErIgHAy8CEatbrB1QAbYFEYIKIdKzheLcCd4vISiAaT6vnvwOovqWqqaqa2qpVq5rPwviE7m1jGNnzPN75ajuHj575k8gVlcqTMzcS1zSc2wfV9E/JGFMXnCwwu/hhK6IdUHWI3GigB7BYRLKBi4AZ3gv9NwJzVLVMVfOAr4HU0x1MVTep6nBV7Qt8CGytszMxPuG+ockcLS3nzSXbznjbj1bkkJFbyKMjuxEeYuONGVMfnCwwK4AkEUkUkRDgemDGiV+qaoGqtlTVBFVNAL4FxqhqGp5usSHiEYmn+Gw63cFEJNb7PQB4DHjDiZMy7kluHc3Vvdvy3jfZ7C+q/dDjBcVlTJm7mf6JzRnZ8zwHExpjqnKswKhqOTAemAtkAP9S1Q0i8pSIjDn91rwGROG5rrICeFdV0wFE5ENgGdBFRHaJyG3ebW4QkS14CtEe4N06PynjunuHJlNaUcnri2vfQP3jwi0UlJTxxOjuNt6YMfVI3Jz/3G2pqamalpbmdgxzhh76ZC3/XruHJQ9dznkxp58cLHNfESNeWcr1F8bzzDU96ymhMQ2biKxU1dNetgB7kt/4oV9fkYSq8uoXmaddT1V5atZGIkMCmTDcpkE2pr5ZgTF+J755BD9OjefjFTvZeaj4lOstyMhjaeYB7h+WTPPIkHpMaIwBKzDGT40f0hkR4c+Lqm/FHC+v4PezN5IUG8VNF3Wo53TGGLACY/xUm5hwftq/PdNW7Sb7wNH/+v3bX21nx8FiHh+dQnCg/TM3xg32yTN+667BnQgOFF5Z+MNWzL7CY7y6KIthKa0ZmGQP0xrjFiswxm/FRodxyyUJfLZmN5n7ir5f/vycTZRXKI+NsvHGjHGTFRjj1+4Y1ImI4ED+uMDTilmdc5jpq3Zz28BEOrSIdDmdMY2bFRjj15pHhnDbpYnMXpfL+t0FTJq5kdjoUO6+vLPb0Yxp9KzAGL9328CONAkL4rb3VrB2Zz4PX9mVqNAgt2MZ0+hZgTF+LyY8mF8O7Mi+wuOcH9+UH50fV/NGxhjH2Z95pkH4xaWJ7Cko4RcDEgmwaZCN8QlWYEyDEBUaxLPX9nI7hjGmCusiM8YY4wgrMMYYYxxhBcYYY4wjrMAYY4xxhBUYY4wxjrACY4wxxhFWYIwxxjjCCowxxhhHiKq6ncE1IrIf2HGWm7cEDtRhHDfZufiehnIeYOfiq87lXDqoao2TLTXqAnMuRCRNVVPdzlEX7Fx8T0M5D7Bz8VX1cS7WRWaMMcYRVmCMMcY4wgrM2XvL7QB1yM7F9zSU8wA7F1/l+LnYNRhjjDGOsBaMMcYYR1iBqSURyRaRdSKyRkTSvMuai8h8Ecn0fm/mds6anOI8JonIbu+yNSIy0u2ctSEiTUVkqohsEpEMEbnYH98TOOW5+N37IiJdquRdIyKFInKfv70vpzkPv3tPAETkfhHZICLrReRDEQkTkUQRWe59Tz4WkZA6P651kdWOiGQDqap6oMqyycAhVX1ORB4GmqnqRLcy1sYpzmMScERVp7iV62yIyHvAUlX9m/fDEQE8ip+9J3DKc7kPP3xfThCRQGA30B+4Gz98X+C/zuMX+Nl7IiJxwFdAiqqWiMi/gM+BkcB0Vf1IRN4A1qrq63V5bGvBnJurgfe8P78H/MjFLI2KiDQBBgFvA6hqqarm44fvyWnOxd9dAWxV1R344ftSRdXz8FdBQLiIBOH54yUXGAJM9f7ekffECkztKTBPRFaKyO3eZa1VNRfA+z3WtXS1V915AIwXkXQRecfXuy+8OgL7gXdFZLWI/E1EIvHP9+RU5wL+975UdT3wofdnf3xfTqh6HuBn74mq7gamADl4CksBsBLIV9Vy72q7gLi6PrYVmNoboKoXAFcCd4vIILcDnaXqzuN1oBNwPp5/gC+6mK+2goALgNdVtQ9wFHjY3Uhn7VTn4o/vCwDebr4xwCduZzkX1ZyH370n3iJ4NZAItAUi8Xz+T1bn10uswNSSqu7xfs8DPgX6AftEpA2A93ueewlrp7rzUNV9qlqhqpXAX/Gcm6/bBexS1eXe11Px/E/a794TTnEufvq+nHAlsEpV93lf++P7Aiedh5++J0OB7aq6X1XLgOnAJUBTb5cZQDtgT10f2ApMLYhIpIhEn/gZGA6sB2YAt3hXuwX4tzsJa+dU53Hig+91DZ5z82mquhfYKSJdvIuuADbiZ+8JnPpc/PF9qeIGftit5Hfvi9cPzsNP35Mc4CIRiRAR4f8+K18A47zrOPKe2F1ktSAiHfH8tQ+e7ox/quozItIC+BfQHs+beJ2qHnIpZo1Ocx7/wNPkVyAbuONEf7kvE5Hzgb8BIcA2PHf4BOBH78kJpziXP+Gf70sEsBPoqKoF3mV+9VmBU56Hv35WngR+ApQDq4H/h+eay0dAc++ym1T1eJ0e1wqMMcYYJ1gXmTHGGEdYgTHGGOMIKzDGGGMcYQXGGGOMI6zAGGOMcYQVGGOMMY6wAmOMDxGRv4vIuJrXNMb3WYExxhjjCCswxtRARBK8k4D91Ttp0zwRCReRxSKS6l2npXeuHUTk5yLymYjMFJHtIjJeRB7wjpT8rYg0r+Vx+4rIl96Rr+dWGcvrlyKyQkTWisg07xAgMeKZTC7Au06EiOwUkWAR6SQic7z7WSoiXb3rXOedgGqtiCxx5D+eadSswBhTO0nAa6raHcgHxtawfg/gRjyDIT4DFHtHSl4G/Kymg4lIMPBnYJyq9gXe8e4HPJNEXaiqvYEM4DbvUCZrgcu864wG5noHN3wLuMe7nweBv3jXeRz4H+9+xtSUyZgzFVTzKsYYPKPRrvH+vBJIqGH9L1S1CCgSkQJgpnf5OqBXLY7XBU+Rmu8Zn5BAPMPDA/QQkd8DTYEoYK53+cd4xpv6As8cJn8RkSg8I+d+4t0PQKj3+9fA370zHE6vRSZjzogVGGNqp+oggBVAOJ6BA0/0AoSdZv3KKq8rqd3nToANqnpxNb/7O/AjVV0rIj8HBnuXzwCe9XbB9QUW4Zn7I19Vzz95J6p6p4j0B0YBa0TkfFU9WItsxtSKdZEZc/ay8fyPHP5v2PO6shloJSIXg6fLTES6e38XDeR6u9F+emIDVT0CfAe8AszyzltSCGwXkeu8+xER6e39uZOqLlfVx4EDQHwdn4Np5KzAGHP2pgB3icg3QMu63LGqluIpWs+LyFpgDZ6uLoDfAcuB+cCmkzb9GLjJ+/2EnwK3efezAc/shgAviMg6EVkPLMFzDceYOmPD9RtjjHGEtWCMMcY4wi7yG+MCEXkNGHDS4ldU9V038hjjBOsiM8YY4wjrIjPGGOMIKzDGGGMcYQXGGGOMI6zAGGOMcYQVGGOMMY74/yXXaKb2bC5pAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "train_means = grid_search.cv_results_['mean_train_score']\n",
    "train_stds = grid_search.cv_results_['std_train_score']\n",
    "\n",
    "n_leafs = len(num_leaves_s)\n",
    "\n",
    "x_axis = num_leaves_s\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot(x_axis, -test_means)\n",
    "plt.xlabel('num_leaves')\n",
    "plt.ylabel('Log loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T05:39:55.615440Z",
     "start_time": "2020-06-25T05:39:55.612447Z"
    }
   },
   "source": [
    "### 叶子节点的最小样本数min_child_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T07:13:58.347106Z",
     "start_time": "2020-06-25T07:02:47.493256Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  9.3min remaining:  2.3min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 11.2min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=0.7, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=20,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=380,\n",
       "        n_jobs=4, num_class=9, num_leaves=60, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.7, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'min_child_samples': range(20, 70, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score=True,\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "         'objective': 'multiclass',\n",
    "         'num_class': 9,\n",
    "         'n_jobs': 4,\n",
    "         'learning_rate': 0.1,\n",
    "         'n_estimators': n_estimators_1,\n",
    "         'max_depth': 7,\n",
    "         'num_leaves': 60,\n",
    "         'max_bin':127,\n",
    "         'subsample': 0.7,\n",
    "         'bagging_freq':1,\n",
    "         'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent = False, **params)\n",
    "\n",
    "min_child_samples_s = range(20, 70, 10)\n",
    "tuned_parameters = dict(min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, \n",
    "                           n_jobs = 4,\n",
    "                           param_grid = tuned_parameters,\n",
    "                           cv = kfold,\n",
    "                           scoring = \"neg_log_loss\",\n",
    "                           verbose = 5,\n",
    "                           refit = False,\n",
    "                           return_train_score = True,\n",
    "                          )\n",
    "grid_search.fit(x_train, y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T07:14:34.294334Z",
     "start_time": "2020-06-25T07:14:34.290350Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.474760947542\n",
      "{'min_child_samples': 40}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T07:14:37.324620Z",
     "start_time": "2020-06-25T07:14:37.247831Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAD8CAYAAABZ/vJZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8FVX6+PHPk0oLhJDQAwm9SgtFKWKDqIgNRURWrGtBdMW2X9ff+pX97irqIirrir0gCNgQFVYFpEgLGJBOCAECIqFFIJCQ5Pn9cedmLzEhl5BkUp7363VfuXPmzJnnDtw8OXNm5oiqYowxxpytALcDMMYYUzFZAjHGGFMslkCMMcYUiyUQY4wxxWIJxBhjTLFYAjHGGFMslkCMMcYUiyUQY4wxxWIJxBhjTLEEuR1AaYqMjNSYmBi3wzDGmApl9erVB1Q1qqh6lTqBxMTEkJCQ4HYYxhhToYjITn/q2SksY4wxxeJXAhGReBHZIiJJIvLEGeoNExEVkThneaSIJPq8ckWkq7MuRESmiMhWEdksItc75RN96m8VkSM+7ef4rJt9bh/dGGPMuSjyFJaIBAKTgcuAVGCViMxW1Y356oUBY4EV3jJVnQpMddZ3Br5Q1URn9ZPAflVtIyIBQISzzZ982nwA6OazmxOq2vWsP6UxxpgS508PpBeQpKrJqpoFTAeuLqDeeGACcLKQdkYA03yWbwf+AaCquap6wI9tjDHGlBP+JJAmwG6f5VSnLI+IdAOiVXXOGdoZjpMMRCTcKRsvImtEZKaINMjXZnMgFpjvU1xNRBJEZLmIXFPQTkTkbqdOQlpamh8fzxhjTHH4k0CkgLK8Waic008TgXGFNiDSG8hQ1fVOURDQFFiqqt2BZcAL+Ta7CZilqjk+Zc1UNQ64GXhJRFr+LjDVKaoap6pxUVFFXoVmjDGmmPxJIKlAtM9yU2Cvz3IY0AlYKCIpQB9gtncg3XETp5+KOghkAJ85yzOB7vn2m38bVHWv8zMZWMjp4yPGGGPKkD8JZBXQWkRiRSQEzy/2vCugVDVdVSNVNUZVY4DlwFBVTYC8HsoNeMZOvNso8CUw0Cm6BMgblBeRtkBdPD0Tb1ldEQl13kcCfX23KUm5ucrfv97EzoPHS6N5Y4ypFIpMIKqaDYwB5gGbgBmqukFEnhGRoX7sYwCQ6vQafD0OPC0i64BRnH4KbAQwXU+fsL09kCAia4EFwLP5rwQrKSkHjzN95S6umLSYWatTsXnjjTHm96Qy/3KMi4vT4t6JvufICf70cSIrdxxiyHmN+L9rO1OnenAJR2iMMeWPiKx2xpvPyO5EL0ST8OpMu6sPjw5uy9z1+7hi0mJWJB90OyxjjCk3LIGcQWCAcP9Frfjk3gsIDhRGvLGcF+Zt4VROrtuhGWOM6yyB+KFLdDhfje3PsB5NeXVBEsP+vYyUAzbAboyp2iyB+KlmaBAThnVh8s3d2ZF2jCtfXszMhN02wG6MqbIsgZylK89rxNyHBtC5aR0enbWOMdN+Ij3jlNthGWNMmbMEUgyNw6sz9c4+PBbflnnr93H5pEUstwF2Y0wVYwmkmAIDhPsGegbYQ4MDGfHGcp6ft9kG2I0xVYYlkHPUJTqcOQ/048Ye0UxesJ1hr/1oA+zGmCrBEkgJqBkaxHPDzuO1kd1JOZjBFS8vZoYNsBtjKjlLICXo8s6NmPtQf7o0DeexWesY85ENsBtjKi9LICWsUZ3qfHhnbx6Pb8e8DfuIn7SIZdttgN0YU/lYAikFgQHCvQNb8tl9fakeHMjNby7nubk2wG6MqVwsgZSizk3rMGdsP4bHRfPawu1c/9qP7LABdmNMJWEJpJTVCAni2evP49+3dGfnwQyufHkxM1bZALsxpuKzBFJG4jt5Bti7Rofz2CfruG/qGo5kZLkdljHGFJslkDLUqE51PryjN3++vB3fbfqV+JcW8+P2A26HZYwxxeJXAhGReBHZIiJJIvLEGeoNExH1zocuIiNFJNHnlSsiXZ11ISIyRUS2ishmEbneKR8tImk+29zp0/6tIrLNed16bh/dHQEBwh8vbMmn9/alRkggI99cwbPfbCYr2wbYjTEVS5EzEopIILAVuAxIxTNH+oj808mKSBjwFRACjPHOie6zvjPwhaq2cJb/FwhU1b8486ZHqOoBERkNxKnqmHzbRwAJQBygwGqgh6oeLiz2c5mRsCxkZGUzfs5Gpq3cTecmdZh0U1daRNVyOyxjTBVXkjMS9gKSVDVZVbOA6cDVBdQbD0wAThbSzghgms/y7cA/AFQ1V1WLOpczGPhWVQ85SeNbIN6P+MutGiFB/OO68/j3LT3YfTiDK19ewvSVu2yA3RhTIfiTQJoAu32WU52yPCLSDYhW1TlnaGc4TgIRkXCnbLyIrBGRmSLSwKfu9SKyTkRmiUi0v3FUVPGdGjL3wQF0bx7OE5/+zL0f2gC7Mab88yeBSAFleX8iO6efJgLjCm1ApDeQoarrnaIgoCmwVFW7A8uAF5x1XwIxqnoe8B3wnj9x+OzrbhFJEJGEtLS0M36w8qRhnWp8cHtv/ueKdny/2RlgT7IBdmNM+eVPAkkFon2WmwJ7fZbDgE7AQhFJAfoAs70D6Y6bOP301UEgA/jMWZ4JdAdQ1YOqmumUvwH08DMOnO2nqGqcqsZFRUX58fHKj4AA4e4BnjvYa4QGMvKtFfzjm002wG6MKZf8SSCrgNYiEisiIXiSwWzvSlVNV9VIVY1R1RhgOTDUO4ju9FBuwDN24t1G8fQ0BjpFlwAbnfqNfPY9FNjkvJ8HDBKRuiJSFxjklFU6nZrUYc4D/RjRqxmv/5DMda8tZXvaMbfDMsaY0xSZQFQ1GxiD55f1JmCGqm4QkWdEZKgf+xgApKpqcr7yx4GnRWQdMIr/ngIbKyIbRGQtMBYY7cRxCM9A/Srn9YxTVinVCAni79d25vVRPdhz+ARDXl7CNBtgN8aUI0VexluRlffLeP31628nGTdjLUuSDjC4YwOeve486tYMcTssY0wlVZKX8RqXNahdjfdv78WTV7Rn/ub9xE9axFIbYDfGuMwSSAURECDcNaAFn93Xl1qhQdzy1gr+8bUNsBtj3GMJpILxDLD35+ZezXh9UTLX/mspSfttgN0YU/YsgVRA1UMC+b9rOzNlVA/2HjnBkFcW89EKG2A3xpQtSyAV2KCODZn30AB6xkTwP5/9zB8/WM2h43YHuzGmbFgCqeDq167Ge7f14i9XtmfhljTiX1rEkm02wG6MKX2WQCqBgADhzv4t+Oz+C6hdPZhb3lrB/321kczsHLdDM8ZUYpZAKpGOjevw5Zh+3NKnGW8s3sF1//rRBtiNMaXGEkglUz0kkL9d05k3/hDHL+knGfLKYqau2GkD7MaYEmcJpJK6rEMD5j7Yn54xETz52XrutgF2Y0wJswRSiXkH2J8a0oEftqQx+KVFLN5WcR5xb4wp3yyBVHIBAcId/WL5/P6+hFcPZtRbK/nbHBtgN8acO0sgVUSHxrWZPaYfo/o0580lO7h28o8k7T/qdljGmArMEkgVUj0kkPHXdOKtW+PY99tJrnx5CR8stwF2Y0zxWAKpgi5p34C5D/Wnd4t6PPX5eu56fzUHj2UWvaExxviwBFJF1Q+rxruje/LUkA4s2ppG/KTFLNpqA+zGGP/5lUBEJF5EtohIkog8cYZ6w0REvfOhi8hIEUn0eeWKSFdnXYiITBGRrSKyWUSud8ofFpGNIrJORL4XkeY+7ef4tDW74CiMv7wD7F+M8Qyw/+HtlYy3AXZjjJ+KnJFQRAKBrcBlQCqe6WRHqOrGfPXCgK+AEGCMd050n/WdgS9UtYWz/L9AoKr+xZk3PUJVD4jIRcAKVc0QkXuBgao63NnmmKrW8vfDVZYZCcvCyVM5/OPrTby3bCftGobxyohutG4Q5nZYxhgXlOSMhL2AJFVNVtUsYDpwdQH1xgMTgJOFtDMCmOazfDvwDwBVzVXVA877Baqa4dRZDjT1I0ZzjqoFB/K/V3fi7dFxpB3NZMgrS/hgWYoNsBtjCuVPAmkC7PZZTnXK8ohINyBaVeecoZ3hOAlERMKdsvEiskZEZopIgwK2uQP4xme5mogkiMhyEbnGj9jNWbq4XQO+eag/fVrU46kvNnDX+wk2wG6MKZA/CUQKKMv7s9Q5/TQRGFdoAyK9gQxVXe8UBeHpWSxV1e7AMuCFfNvcAsQBz/sUN3O6VTcDL4lIywL2dbeTZBLS0mxQuDjqh1XjndE9+etVHVi07QCDX1rMDzbAbozJx58EkgpE+yw3Bfb6LIcBnYCFIpIC9AFmewfSHTdx+umrg0AG8JmzPBPo7l0pIpcCTwJDVTXvz19V3ev8TAYWAt3yB6uqU1Q1TlXjoqKi/Ph4piABAcJtfWOZPaYvETWDufXtlTzz5UZOnrIBdmOMhz8JZBXQWkRiRSQETzLIuwJKVdNVNVJVY1Q1Bs+4xVDvILrTQ7kBz9iJdxsFvgQGOkWXABud+t2A15029nu3EZG6IhLqvI8E+nq3MaWnXUPPHeyjL4jh7aU7uGbyUrb+anewG2P8SCCqmg2MAeYBm4AZqrpBRJ4RkaF+7GMAkOr0Gnw9DjwtIuuAUfz3FNjzQC1gZr7LddsDCSKyFlgAPJv/SjBTOqoFB/L00I68M7onB45lctUrS3h/mQ2wG1PVFXkZb0Vml/GWvLSjmTw6ay0Lt6Rxcbv6TBh2HpG1Qt0OyxhTgkryMl5j8kSFhfLO6J48fVUHliQdIP6lxSzcsr/oDY0xlY4lEHPWRITRfWP5ckw/6tUMYfQ7q3h69gYbYDemirEEYoqtbcMwvhjTl9EXxPDujyn8/etNbodkjClDlkDMOfEOsN/SpxkfrdjFzoPH3Q7JGFNGLIGYEjH24tYEBQr//Har26EYY8qIJRBTIurXrsbtfWP5InEvG/amux2OMaYMWAIxJeaPF7akTvVgXpi3xe1QjDFlwBKIKTF1qgdz78CWLNiSxorkg26HY4wpZZZATIm69fwYGtQOZcK8LXanujGVnCUQU6KqhwTy4CVtWL3zMN9vshsMjanMLIGYEndDXFNi6tXg+XlbyMm1XogxlZUlEFPiggMDGDeoLVt+PcrstXvcDscYU0osgZhScWXnRnRsXJsX/7OVrOxct8MxxpQCSyCmVAQECI/FtyP18AmmrdzldjjGmFJgCcSUmgGtI+nTIoJX5m/jeGa22+EYY0qYJRBTakQ8vZADx7J4e8kOt8MxxpQwSyCmVHVvVpdBHRowZVEyh49nuR2OMaYE+ZVARCReRLaISJKIPHGGesNEREUkzlke6UxL633likhXZ12IiEwRka0isllErnfKQ0XkY2dfK0Qkxqf9PzvlW0Rk8Ll8cFN2HhncluNZ2bz2w3a3QzHGlKAiE4iIBAKTgcuBDsAIEelQQL0wYCywwlumqlNVtauqdsUz73mKqiY6q58E9qtqG6fdH5zyO4DDqtoKmAg857TfAbgJ6AjEA/9yYjPlXJsGYVzbrSnv/pjCL+kn3A7HGFNC/OmB9AKSVDVZVbOA6cDVBdQbD0wAThbSzghgms/y7cA/AFQ1V1UPOOVXA+8572cBl4iIOOXTVTVTVXcASU5spgJ46NLWoDDpu21uh2KMKSH+JJAmwG6f5VSnLI+IdAOiVXXOGdoZjpNARCTcKRsvImtEZKaINMi/P1XNBtKBev7E4bR9t4gkiEhCWlqaHx/PlIXoiBqM7NOMGQm7Sdp/zO1wjDElwJ8EIgWU5T2fQkQC8JxqGldoAyK9gQxVXe8UBQFNgaWq2h1YBrxQxP7OGEdegeoUVY1T1bioqKjCQjIuuP+iVlQPDuSf39rj3o2pDPxJIKlAtM9yU2Cvz3IY0AlYKCIpQB9gtncg3XETp5++OghkAJ85yzOB7vn3JyJBQB3gkB9xmHIuslYod/Zvwdc/72Pt7iNuh2OMOUf+JJBVQGsRiRWREDzJYLZ3paqmq2qkqsaoagywHBiqqgmQ10O5Ac/YiXcbBb4EBjpFlwAbnfezgVud98OA+U792cBNzlVasUBrYOXZf2Tjpjv7xxJRM4TnbdIpYyq8IhOIMw4xBpgHbAJmqOoGEXlGRIb6sY8BQKqqJucrfxx4WkTW4blCy3sK7C2gnogkAQ8DTzhxbABm4Ek0c4H7VTXHj/2bciSsWjD3X9SKJUkHWJp0oOgNjDHlllTmSX/i4uI0ISHB7TBMPidP5XDxCwuJCgvl8/v74rnIzhhTXojIalWNK6qe3Yluyly14EAeuqwNa1PTmbdhn9vhGGOKyRKIccV13ZrQqn4tnp+3hewce9y7MRWRJRDjiqDAAB4Z1Jbtacf5dI1NOmVMRWQJxLhmcMcGdIkOZ+J3Wzl5yq6HMKaisQRiXCMiPB7fll/ST/Lh8p1uh2OMOUuWQIyrLmgZSf/WkUxekMTRk6fcDscYcxYsgRjXPTa4HYczTvHGYpt0ypiKxBKIcV3npnW4snMj3lyczIFjmW6HY4zxkyUQUy48PKgNmdm5vDo/ye1QjDF+sgRiyoWWUbW4Ma4pU1fsZPehDLfDMcb4wRKIKTfGXtKaABEmfrfV7VCMMX6wBGLKjUZ1qjP6ghg++2kPW/YddTscY0wRLIGYcuXegS2pFRrEC/+xx70bU95ZAjHlSniNEP44oAXfbvyV1TsPux2OMeYMLIGYcue2vrFE1grlubmbqczTDRhT0VkCMeVOzdAgxl7SipU7DvHD1jS3wzHGFMKvBCIi8SKyRUSSROSJM9QbJiLqnQ9dREaKSKLPK1dEujrrFjptetfVd8on+pRtFZEjPu3n+KybXXAUpjK4qWczoiOqM2HuFnJzrRdiTHkUVFQFEQkEJgOXAanAKhGZraob89ULA8YCK7xlqjoVmOqs7wx8oaqJPpuN9M6d7rPNn3zafADo5rP6hKp29fOzmQosJCiAcZe15aGPE5nz8y8M7dLY7ZCMMfn40wPpBSSparKqZgHTgasLqDcemACcLKSdEcC0s4yvONuYSmJol8a0axjGi//ZwimbdMqYcsefBNIE2O2znOqU5RGRbkC0qs45QzvD+X0yeMc5HfWU5JsYW0SaA7HAfJ/iaiKSICLLReQaP2I3FVhAgPBYfFt2Hszg41W7i97AGFOm/EkgUkBZ3klpEQkAJgLjCm1ApDeQoarrfYpHqmpnoL/zGpVvs5uAWarqO9NQM2ei95uBl0SkZQH7uttJMglpaTYAW9Fd1LY+PWPq8vL32ziRZZNOGVOe+JNAUoFon+WmwF6f5TCgE7BQRFKAPsBs70C64yby9T5UdY/z8yjwEZ5TZRSxzV7nZzKwkNPHR7x1pqhqnKrGRUVF+fHxTHkmIjwW3479RzN598cUt8MxxvjwJ4GsAlqLSKyIhOD5xZ53BZSqpqtqpKrGqGoMsBwY6h0cd3ooN+AZO8EpCxKRSOd9MDAEWO+zvi1QF1jmU1ZXREKd95FAX+C0gXxTOfWMieDidvV5bWES6Rk26ZQx5UWRCURVs4ExwDxgEzBDVTeIyDMiMtSPfQwAUp1eg1coME9E1gGJwB7gDZ/1I4DpevpdZO2BBBFZCywAns1/JZipvB4d3Jajmdn8e9F2t0MxxjikMt/pGxcXpwkJCUVXNBXCQ9N/Yu6Gffzw6EU0qF3N7XCMqbREZLUz3nxGdie6qTAevqwt2TnKy99vczsUYwyWQEwF0qxeDW7u3YyPV+0m5cBxt8MxpsqzBGIqlDEXtyI4MIB/fmuTThnjNksgpkKpH1aN2/vFMHvtXjbsTXc7HGOqNEsgpsK5e0BL6lQP5vl5NumUMW6yBGIqnDrVg7lvYEsWbkljefJBt8MxpsqyBGIqpFsviKFB7VAm2KRTxrjGEoipkKoFB/LQpW1Ys+sI323a73Y4xlRJlkBMhXVDj6a0iKzJ8/M2k2OTThlT5iyBmAorKDCAcYPasvXXY3yRuMftcIypciyBmArt8k4N6dSkNv/8diuZ2fa4d2PKkiUQU6EFBAiPDW5H6uETTFuxy+1wjKlSLIGYCq9/60jOb1GPV+YncSwz2+1wjKkyLIGYCs8z6VRbDh7P4u0lO9wOx5gqwxKIqRS6NavL4I4NmLIomUPHs9wOx5gqwRKIqTQeGdSWjKxs/rUgye1QjKkS/EogIhIvIltEJElEnjhDvWEiot750EVkpIgk+rxyRaSrs26h06Z3XX2nfLSIpPmU3+nT/q0iss153XpuH91UNq0bhHF996a8v3wne4+ccDscYyq9IhOIiAQCk4HLgQ7ACBHpUEC9MGAssMJbpqpTVbWrqnYFRgEpqpros9lI73pV9b2d+GOf8jed9iOAvwK9gV7AX0Wk7tl+YFO5PXRZG1CY9J1NOmVMafOnB9ILSFLVZFXNAqYDVxdQbzwwAThZSDsjgGnFitJjMPCtqh5S1cPAt0D8ObRnKqEm4dW5pU9zZq7eTdL+Y26HY0yl5k8CaQLs9llOdcryiEg3IFpV55yhneH8PoG845ymekpExKf8ehFZJyKzRCTa3ziMAbj/opZUDw7kxf/Y496NKU3+JBApoCzvwUMiEgBMBMYV2oBIbyBDVdf7FI9U1c5Af+c1yin/EohR1fOA74D3/InDZ193i0iCiCSkpaUV/qlMpVWvVih3DWjBN+v3sXb3EbfDMabS8ieBpALRPstNgb0+y2FAJ2ChiKQAfYDZ3oF0x03k632o6h7n51HgIzynylDVg6qa6VR7A+jhZxzedqeoapyqxkVFRfnx8UxldGf/FkTUDGHCvM1uh2JMpeVPAlkFtBaRWBEJwZMMZntXqmq6qkaqaoyqxgDLgaGqmgB5PZQb8Iyd4JQFiUik8z4YGAKsd5Yb+ex7KLDJeT8PGCQidZ3B80FOmTG/Uys0iDEXtWJp0kGWbDvgdjjGVEpFJhBVzQbG4PllvQmYoaobROQZERnqxz4GAKmqmuxTFgrME5F1QCKwB09vA2CsiGwQkbV4ruoa7cRxCM9A/Srn9YxTZkyBRvZpRpPw6kyYZ5NOGVMapDJ/seLi4jQhIcHtMIyLZq1O5ZGZa3ltZHcu79yo6A2MMYjIalWNK6qe3YluKrVruzWhdf1aPP+fLWTn5LodjjGViiUQU6kFBgiPDG5LctpxPlmT6nY4xlQqlkBMpTeoQwO6Rofz0nfbOHnKJp0ypqRYAjGVnojweHw7fkk/yQfLdrodjjGVhiUQUyWc37IeA9pEMXlhEr+dPOV2OMZUCpZATJXx2OC2HMk4xZuLkouubIwpkiUQU2V0alKHIec14s0lO0g7mln0BsaYM7IEYqqUcYPakpmdy2SbdMqYc2YJxFQpsZE1uTEumqkrdrL7UIbb4RhToVkCMVXOg5e0JkCEid9udTsUYyo0SyCmymlYpxqj+8bwWeIeNu/7ze1wjKmwLIGYKuneC1tSKzSIF+bZpFPGFJclEFMlhdcI4Z4LW/Ldpv0kpNhDnY0pDksgpsq6rW8MUWGhTJi7xR73Xs6t35POD1vTOGo3gZYrQW4HYIxbaoQEMfbiVjz1xQYWbk3jorb13Q7J5HMqJ5eXvtvKvxZuRxUCBNo3qk3PmAh6xUbQMyaCqLBQt8Ossmw+EFOlZWXncuk/f6BmaBBfPdCPgABxOyTj2H0ogwen/8SaXUcYHhfNkC6NSEg5zKqUQ6zZdZiTpzyP54+NrEnPmLr0iq1Hr5gIoiOqI2L/jufC3/lArAdiqrSQoADGDWrDg9MT+XLdXq7u2sTtkAzw1bpfeOLTdaDwyohuXNWlMQD9W0cBnsS/fm86q3YcYlXKIeZt+JUZCZ7H9TeoHXpaD6VtgzD7w6CU+NUDEZF4YBIQCLypqs8WUm8YMBPoqaoJIjISeNSnynlAd1VNFJGFQCPghLNukKruF5GHgTuBbCANuF1Vdzrt5wA/O/V3qeoZp9S1HojxR26ucsXLi8nIyuG7hy8kJMiGBt1yIiuHZ+ZsYNrK3XSNDueVEd2IjqhR5Ha5ucq2/cdYmXKIVTsOsXLHIfb9dhKA2tWCiIuJcJJKXTo3Cbd/4yL42wMpMoGISCCwFbgMSMUzH/kIVd2Yr14Y8BUQAoxR1YR86zsDX6hqC2d5IfBIAfUuAlaoaoaI3AsMVNXhzrpjqlqrqA/lZQnE+GvB5v3c9u4qxl/TiVF9mrsdTpW0ed9vPPDRT2zbf4x7LmzJuEFtCA4s3i96VSX18AlWOj2UlSmHSE47DkC14AC6RofTKyaCnrERdG9Wl5qhdjLGV0mewuoFJKlqstPwdOBqYGO+euOBCcAjhbQzAphW1M5UdYHP4nLgFj9iNOacDGwbRa+YCF7+fhvXd29CjRD7hVJWVJUPV+zib3M2Urt6MB/c0SvvVFVxiQjRETWIjqjB9T2aAnDgWCYJKYdYucMzjvLqgiRy53tmrezY2DMw73nVpV4tG5j3hz/fkibAbp/lVKC3bwUR6QZEq+ocESksgQzHk3h8veOclvoE+Jv+vjt0B/CNz3I1EUnAc3rrWVX9PP9ORORu4G6AZs2anfGDGeMlIjwW35Zh/17Guz+mcN/AVm6HVCUcycji8U/WMW/Dr1zYJooXb+xCZCn98o6sFUp8p0bEd2oEwNGTp1iz64jnlFfKIT5YvpO3luwAoFX9WnmnvHrGRNC0btGn0aoifxJIQaNPeb/oRSQAmAiMLrQBkd5Ahqqu9ykeqap7nFNfnwCjgPd9trkFiAMu9NmmmaruFZEWwHwR+VlVt58WmOoUYAp4TmH58fmMASAuJoJL29fn3wu3M7JXc+rUCHY7pEptVcohHpz2E2nHMnnyivbc0S+2TAe7w6oFc2GbKC5s4+ntZGbn8HNqet44ypx1e5m2chcAjetUo2dsRN7gfKuoWjYwj38JJBWI9lluCuz1WQ4DOgELnUvnGgKzRWSoz/jGTeQ7faWqe5yfR0XkIzynyt4HEJFLgSeBC1U102ebvc7PZGcMpRtwWgIx5lw8Mrgtl09azGs/bOeJy9u5HU6llJOrTF6QxEvfbSU6ogaf3HsB5zUNdzssQoMCiYuJIC4mAgZ64tyy7ygrdxxkVcphftx+kC8SPb/66tYIJi4mIm8cpWOhkxHHAAATrUlEQVTj2sUer6nI/Ekgq4DWIhIL7MGTDG72rlTVdCDSu5x/cNzpodwADPCpEwSEq+oBEQkGhgDfOeu6Aa8D8aq632ebunh6MZkiEgn0xTPmYkyJadewNtd0bcI7S3cw+oIYGtap5nZIlcq+9JM89PFPLE8+xDVdGzP+mk6EVSufPb3AAKFD49p0aFyb0X1jUVV2HszI66GsSjnEtxt/BaB6cCDdm4d7eigxEXRrVpfqIYEuf4LSV2QCUdVsERkDzMNzGe/bqrpBRJ4BElR1dhFNDABSvYPwjlBgnpM8AvEkjzecdc8DtYCZTo/Ge7lue+B1EcnF8wiWZ/NfCWZMSfjTpW2Ys24vL8/fxt+v7ex2OJXGtxt/5dFZa8nKzuWFG7pwffcmFeqGPxEhJrImMc6cMgD7fzvJqpTDrNxxkJUph5n0/TZUIShA6NSkDr2d015xMXUJrxHi8icoeXYnujEF+OsX6/lwxS6+e/hCYiNruh1OhXbyVA7PfrOZd39MoWPj2rwyohstovy+Gr9CST9xijU7D+f1UtalppOV47ljvm2DMHo6g/K9YiNoVKe6y9EWrsTuA6nILIGY4ko7msmFzy/g4nb1efXm7m6HU2FtTzvGAx/9xMZffuO2vjE8cXk7QoMq/6kdr5Oncli7+wgrnSu91uw8zPGsHACa1q2eN4bSMyaCllE1y02PzB5lYsw5iAoL5Y5+sbwyP4l7LkynU5M6bodUoagqM1en8tcvNlAtOIC3bo3jkvYN3A6rzFULDqR3i3r0blEPgOycXDb9cjSvh/LD1jQ+/WkPAPVqhnjuQ4n1jKO0bxRGUDkfmLceiDGF+O3kKQZMWECXpuG8d3svt8OpMI6ePMWTn61n9tq99GkRwUvDu9nFCIVQVZIPHM97/MrKlEOkHvY83almSCDdm9fN66V0jQ6nWnDZ9N6sB2LMOapdLZj7Brbk719vZtn2g5zfsp7bIZV7ibuPMHbaT+w5coJHBrXh3oGtCLT7JQolIrSMqkXLqFrc1Mtz4/Mv6f99BMuqHYd58dutAIQEBtC5aR16OT2U7s3rUqe6u1ewWQ/EmDM4eSqHgc8vpFF4NT6994Jyc466vMnNVd5YnMzz87bQoHY1Xh7RlR7NI9wOq1I4kpGV9xj7FTsOsX5POtm5iojnsvNeMXXzTnvVr10yPT0bRMcSiCkZ01fu4olPf2bKqB4M6tjQ7XDKnbSjmYybuZZFW9O4vFNDnr3uPLuLvxRlZGWTuOuIZxwl5RBrdh7hxCnPwHzzejXy7kXpFRtBTDGvILQEgiUQUzKyc3IZNHERgQHC3IcG2CkZH4u2pvHwjESOnszm/13VgZt7NbNeWhk7lZPLhr2/5T3Ta1XKIY5knKJj49p8NbZ/sdq0MRBjSkhQYACPDG7LfVPX8NlPexjmPN21KsvKzuXF/2zh9UXJtGlQi6l39qFtwzC3w6qSggM9j6fvGh3OXQNakJurJKUd47cTpT9/vCUQY/xweaeGdG5Sh4nfbuWqLo2q1L0M+e06mMED039i7e4j3Ny7GU9d2aFKPLajoggIENo0KJtkXr4vMjamnBARHo9vx54jJ/hoxS63w3HNF4l7uOLlxexIO8ZrI7vz92s7W/KowiyBGOOnfq0juaBlPV6dn8SxzGy3wylTGVnZPDpzLQ9OT6RtwzC+frA/l3du5HZYxmWWQIw5C4/Ft+Pg8SzeWrzD7VDKzIa96Qx5ZQmz1qTywMWt+PjuPjbBkgEsgRhzVrpGhxPfsSFvLE7m4LHMojeowFSVd5fu4NrJP3I8M5upd/Zm3KC25f7xGqbs2P8EY87SI4PbkJGVzb8WVt65zA4fz+Ku91fz9Jcb6d86km8eHMAFLSOL3tBUKZZAjDlLreqHMaxHUz5YtpM9R064HU6JW7b9IJdPWsyirWn8vyEdePPWOCJqVr65LMy5swRiTDE8eGkbEJj03Va3Qykx2Tm5/PM/W7j5zeVUDwnk0/su4PZ+sXZjoCmUXwlEROJFZIuIJInIE2eoN0xEVETinOWRIpLo88oVka7OuoVOm9519Z3yUBH52NnXChGJ8Wn/z075FhEZfC4f3Jhz0SS8On/o05xZq1NJ2n/U7XDO2Z4jJxjxxnJenp/Edd2aMueBfvYIe1OkIhOIiAQCk4HLgQ7ACBHpUEC9MGAssMJbpqpTVbWrqnYFRgEpqpros9lI73qf+c/vAA6raitgIvCc034HPPOxdwTigX85sRnjivsuakWNkCBemFexeyFz1+/jikmL2bj3N14a3pUXb+xCzVC7x9gUzZ8eSC8gSVWTVTULmA5cXUC98cAE4GQh7YwApvmxv6uB95z3s4BLxNOHvhqYrqqZqroDSHJiM8YVETVDuKt/C+Zu2Efi7iNuh3PWTp7K4S+f/8w9H66meb0afDW2P9d0a+J2WKYC8SeBNAF2+yynOmV5RKQbEK2qc87QznB+n0DecU5fPSX/PdGatz9VzQbSgXr+xGFMWbujfyz1aobw3DebqUgPJt3261GumbyUD5fv4q7+scy654JiP7nVVF3+JJCCRtDyvikiEoDnVNO4QhsQ6Q1kqOp6n+KRqtoZ6O+8RhWxvzPG4bOvu0UkQUQS0tLSCgvJmBJRKzSIMRe3YlnyQZYkHXA7nCKpKh+t2MVVry4h7Wgm79zWkyev7EBIkF1PY86eP/9rUoFon+WmwF6f5TCgE7BQRFKAPsBs70C64yby9T5UdY/z8yjwEf89HZW3PxEJAuoAh/yIw9vuFFWNU9W4qKgoPz6eMefm5t7NaBJenQlzt5TrXkj6iVOM+egn/uezn4lrHsE3D/Xnorb13Q7LVGD+JJBVQGsRiRWREDzJYLZ3paqmq2qkqsaoagywHBiqqgmQ10O5Ac/YCU5ZkIhEOu+DgSGAt3cyG7jVeT8MmK+eb+Vs4CbnKq1YoDWwspif25gSExoUyMOXteHnPel8s36f2+EUaPXOw1wxaTHzNuzj8fh2vH97L+qH2Tzl5twUmUCccYgxwDxgEzBDVTeIyDMiMtSPfQwAUlU12acsFJgnIuuARGAP8Iaz7i2gnogkAQ8DTzhxbABmABuBucD9qprjx/6NKXXXdGtCmwa1eGHeFrJzct0OJ09OrjJ5QRI3vr4MEZhxz/ncO7AlATYplikBNiOhMSXk242/ctf7CTx7XWdu6tXM7XD49beTPDwjkaVJBxlyXiP+fl1nalezqWZN0WxGQmPK2KXt69O9WTgvfbeNa7o1oVqwe7cpLdi8n3Ez15KRlc1z13fmxrhou6PclDi79MKYEiIiPBbfjn2/neT9ZSmuxJCZncP4ORu57d1V1A8LZc4D/Rje0+YpN6XDEogxJahPi3pc2CaKyQu2k14Gc1L72nHgONe/9iNvLdnBH85vzuf396VVfZun3JQeSyDGlLBHB7cl/cQp3liUXHTlEvLpmlSGvLyY3YdO8PqoHjxzdSdXT6GZqsESiDElrFOTOlzVpTFvLdnB/qOFPdmnZBzLzOZPHyfy8Iy1dGxch28e7M/gjg1LdZ/GeFkCMaYUjLusDadycpk8P6nU9vFzajpDXl7MF4l7eOjS1nx0V28ah1cvtf0Zk58lEGNKQUxkTYb3jOajlbvYdTCjRNvOzVXeXJzMda8tJTM7l2l39eGhS9vYVLOmzNn/OGNKydhLWhMYIEwswUmnDhzL5I73VvG3rzYxsG19vh7bn94t6pVY+8acDUsgxpSSBrWrMfqCWD5P3MOmX3475/aWJh3g8kmLWbr9IM9c3ZEpo3pQ16aaNS6yBGJMKbr3wpaEhQbxwrwtxW7jVE4uE+Zu5pa3VlC7WhCf39eXP5wfY/d2GNdZAjGmFNWpEcw9A1vy/eb9rEo5dNbb7z6UwY2vL+NfC7dzY49ovnygHx0a1y6FSI05e5ZAjCllt10QS/2wUCbMPbtJp75a9wtXvLyYpF+P8cqIbjw37DxqhNjTh0z5YQnEmFJWPSSQsZe0ZlXKYRZuKXqSsxNZOfz503Xc/9EaWkbV4usH+3NVl8ZlEKkxZ8cSiDFlYHjPaJrXq8FzczeTm1t4L2Tzvt+46tUlTFu5m3subMnMe84nOqJGGUZqjP8sgRhTBoIDA3j4sjZs3neUL9f9biJNVJUPlu9k6KtLST9xig/u6MUTl7cj2O7tMOWY/e80poxcdV5j2jeqzYv/2UpW9n8nnTqSkcU9H67mqc/Xc36LenzzYH/6t7bpmE3551cCEZF4EdkiIkki8sQZ6g0TEfXOhy4iI0Uk0eeVKyJd820zW0TW+yx/7FM/RUQSnfIYETnhs+7fxfvIxrgjIEB4LL4tuw5l8PGqXQCs3HGIKyYtZv7m/Tx5RXveGd2TyFqhLkdqjH+KvKRDRAKBycBlQCqwSkRmq+rGfPXCgLHACm+Zqk4FpjrrOwNfqGqizzbXAcd821HV4T7rXwTSfVZvV9XTEpAxFcnANlH0io1g0vdJpB3L4tX524iOqMEn917AeU3D3Q7PmLPiTw+kF5CkqsmqmgVMB64uoN54YAJQ2ONHRwDTvAsiUgvPnOd/K6iyeO6SutF3G2MqOhHh8fi2HDiWycvfb2Nol8bMeaCfJQ9TIflzUXkTYLfPcirQ27eCiHQDolV1jog8Ukg7wzk98YwHXgQKe9Jcf+BXVd3mUxYrIj8BvwF/UdXFfsRvTLnSo3kEf7myPVFhoQzt0tjuKDcVlj8JpKD/3XnXIYpIADARGF1oAyK9gQxVXe8sdwVaqeqfRCSmkM1O67EAvwDNVPWgiPQAPheRjqp62kOGRORu4G6AZs2anfmTGeOSO/u3cDsEY86ZP6ewUoFon+WmgO91iGFAJ2ChiKQAfYDZ3oF0x02cngzOB3o49ZcAbURkoXeliAQB1wEfe8tUNVNVDzrvVwPbgTb5g1XVKaoap6pxUVF2JYsxxpQWfxLIKqC1iMSKSAieZDDbu1JV01U1UlVjVDUGWA4MVdUEyOuh3IBn7MS7zWuq2tip3w/YqqoDffZ5KbBZVVO9BSIS5QzoIyItgNZA2c0Zaowx5jRFnsJS1WwRGQPMAwKBt1V1g4g8AySo6uwzt8AAIFVVz+aXff4ei7edZ0QkG8gB7lHVs386nTHGmBIhZ/Nwt4omLi5OExIS3A7DGGMqFBFZrapxRdWzO9GNMcYUiyUQY4wxxWIJxBhjTLFYAjHGGFMslXoQXUTSgJ3n0EQkcKCEwilJFtfZsbjOjsV1dipjXM1Vtcgb6Sp1AjlXIpLgz5UIZc3iOjsW19mxuM5OVY7LTmEZY4wpFksgxhhjisUSyJlNcTuAQlhcZ8fiOjsW19mpsnHZGIgxxphisR6IMcaYYrEEAohItIgsEJFNIrJBRB50yiNE5FsR2eb8rFtO4npaRPb4zA9/RRnHVU1EVorIWieu/3XKY0VkhXO8Pnae3lwe4npXRHb4HC9XpkUWkUAR+UlE5jjLrh6vM8Tl+vESkRQR+dnZv/fJ3q5+H88Ql6vfRyeGcBGZJSKbnd8X55fF8bIE4pENjFPV9njmM7lfRDoATwDfq2pr4HtnuTzEBTBRVbs6r6/LOK5M4GJV7QJ0BeJFpA/wnBNXa+AwcEc5iQvgUZ/jlVjGcXk9CGzyWXb7eHnljwvKx/G6yNm/91JUt7+PhcUF7n4fASYBc1W1HdAFz79nqR8vSyCAqv6iqmuc90fxHPwmeKbgfc+p9h5wTTmJy1XqccxZDHZeClwMzHLK3ThehcXlOhFpClwJvOksCy4fr4LiKudc/T6WVyJSG890F28BqGqWqh6hDI6XJZB8nCl2uwErgAaq+gt4fpkD9ctJXABjRGSdiLztUlc+UEQSgf3At3hmiDyiqtlOlVRcSHb541JV7/H6P+d4TRSR0LKOC3gJeAzIdZbrUQ6OVwFxebl9vBT4j4isFs801VA+vo8FxQXufh9bAGnAO86pyDdFpCZlcLwsgfgQkVrAJ8BD+edad1MBcb0GtMRzmuYX4MWyjklVc1S1K54pjnsB7QuqVrZR/T4uEekE/BloB/QEIoDHyzImERkC7HemYs4rLqBqmR6vQuICl4+Xo6+qdgcux3PqdoALMRSkoLjc/j4GAd2B11S1G3CcMjq9ZwnEISLBeH5JT1XVT53iX0WkkbO+EZ6/al2PS1V/dX5R5gJv4PkF7gqnq7wQzxhNuHjmswfPL/C95SCueOdUoKpqJvAOZX+8+gJDRSQFz9TOF+P5y9/t4/W7uETkw3JwvFDVvc7P/cBnTgyufx8LiqscfB9T8cz66u1tz8KTUEr9eFkCIe989FvAJlX9p8+q2cCtzvtbgS/KQ1ze/xSOa4H1ZRxXlIiEO++r45nDfhOwABjmVHPjeBUU12afL5HgOQ9cpsdLVf+sqk1VNQbPdM3zVXUkLh+vQuK6xe3jJSI1RSTM+x4Y5MTg9vexwLjc/j6q6j5gt4i0dYouATZSBseryDnRq4i+wCjgZ+f8OcD/AM8CM0TkDmAXcEM5iWuEc2mlAinAH8s4rkbAeyISiOePkBmqOkdENgLTReRvwE84g3rlIK75IhKF57RRInBPGcdVmMdx93gVZqrLx6sB8JknfxEEfKSqc0VkFe5+HwuL6wOXv48AD+D5dwsBkoHbcL4DpXm87E50Y4wxxWKnsIwxxhSLJRBjjDHFYgnEGGNMsVgCMcYYUyyWQIwxxhSLJRBjjDHFYgnEGGNMsVgCMcYYUyz/H+ZLOwrKOCnTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "train_means = grid_search.cv_results_['mean_train_score']\n",
    "train_stds = grid_search.cv_results_['std_train_score']\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "plt.plot(x_axis, -test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 每棵树的行采样比例sub_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T07:45:23.626166Z",
     "start_time": "2020-06-25T07:35:29.292771Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  8.3min remaining:  2.1min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed:  9.9min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=0.7, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=40,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=380,\n",
       "        n_jobs=4, num_class=9, num_leaves=60, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'subsample': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score=True,\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "         'objective': 'multiclass',\n",
    "         'num_class': 9,\n",
    "         'n_jobs': 4,\n",
    "         'learning_rate': 0.1,\n",
    "         'n_estimators': n_estimators_1,\n",
    "         'max_depth': 7,\n",
    "         'num_leaves': 60,\n",
    "         'min_child_samples': 40,\n",
    "         'max_bin': 127,\n",
    "         'bagging_freq': 1,\n",
    "         'colsample_bytree': 0.7,\n",
    "         }\n",
    "\n",
    "lg = LGBMClassifier(silent = False, **params)\n",
    "#设置sub_sample_s参数的调优范围\n",
    "sub_sample_s = [i / 10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict(subsample = sub_sample_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, \n",
    "                           n_jobs = 4,\n",
    "                           param_grid = tuned_parameters,\n",
    "                           cv = kfold,\n",
    "                           scoring = 'neg_log_loss',\n",
    "                           verbose = 5,\n",
    "                           refit = False,\n",
    "                           return_train_score = True,\n",
    "                          )\n",
    "grid_search.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T07:45:32.391320Z",
     "start_time": "2020-06-25T07:45:32.388328Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.474302682\n",
      "{'subsample': 0.8}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T07:45:37.537784Z",
     "start_time": "2020-06-25T07:45:37.461953Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VfW57/HPk5khTBkYAwlJAJkMEAKCMqmV1lM4VixQiiAqTsC5rZ29516P3Hvr8fQcW5VqUUG0FMWhR6SDViYHxoCATIEQIgSQDEAYQsj03D/2Stxihh1IsnZ2nvfrlRd7r/FZ27i/+a3fWusnqooxxhhTkyC3CzDGGOPfLCiMMcbUyoLCGGNMrSwojDHG1MqCwhhjTK0sKIwxxtTKgsIYY0ytLCiMMcbUyoLCGGNMrULcLqAhREdHa3x8vNtlGGNMs7J9+/Z8VY2pazmfgkJEJgK/A4KBl1T1yRqWmwK8CQxX1XQRmQH81GuRwcBQVd0pItOBXwEKnAB+qKr5IvI4cD+Q56zzK1X9a231xcfHk56e7suhGGOMcYjIF74sV+epJxEJBhYB3wb6A9NFpH81y0UCC4AtldNUdbmqpqhqCjATyHZCIgRP8IxX1cHAbmCe1+aerlyvrpAwxhjTuHzpo0gDMlU1S1VLgNeBydUstxB4CiiuYTvTgRXOa3F+2oiIAO3wtCqMMcb4GV+CojtwzOt9jjOtiogMAeJUdXUt25mKExSqWgo8BHyOJyD6Ay97LTtPRHaLyBIR6VjdxkRkroiki0h6Xl5edYsYY4xpAL4EhVQzrerZ5CISBDwNPFrjBkRGAEWqusd5H4onKIYA3fCcevqls/jzQCKQApwE/rO6barqYlVNVdXUmJg6+2KMMcZcJV+CIgeI83rfg6+fJooEBgLrRSQbGAmsEpFUr2Wm8dVpJ/CEAKp6WD0DYqwERjnTTqlquapWAC/iOfVljDHGJb4ExTYgWUQSRCQMz5f+qsqZqlqoqtGqGq+q8cBmYJKqpkNVi+MuPH0blY4D/UWksilwK7DfWb6r13J3AHuu6siMMcY0iDovj1XVMhGZB7yP5/LYJaq6V0SeANJVdVXtW2AMkKOqWV7bPCEi/wZ8JCKlwBfAbGf2UyKSguf0VjbwQD2PyRhjTAOSQBgKNTU1Va/mPoo9xwtZvfskP5/YF8/FV8YY03KIyHZVTa1ruRb9CI8dR8/wwobDbMoqcLsUY4zxWy06KL6fGkdsZDjPrDnkdinGGOO3WnRQRIQG88DYRDZnnWbrkdNul2OMMX6pRQcFwA/SehLdNoxn11qrwhhjqtPig6JVWDD339Sbjw/ls+PoGbfLMcYYv9PigwLghyN70bF1KM9aX4UxxnyDBQXQJjyE+27qzbqMPD7PKXS7HGOM8SsWFI67b+hFu4gQ66swxpgrWFA4IiNCmXNjAh/sO8X+k+fcLscYY/yGBYWXe0Yl0DY8hOfWZrpdijHG+A0LCi/tW4cye1Q8f91zkkOnzrtdjjHG+AULiivMuTGBVqHBPLfOWhXGGAMWFN/QqU0YM2/oxXu7TpCVd8HtcowxxnUWFNW4/6behIUEsWjdYbdLMcYY11lQVCO6bTgzRvTiv3ce52hBkdvlGGOMq3wKChGZKCIZIpIpIr+oZbkpIqKVw6CKyAwR2en1U+EMSoSITBeRz0Vkt4j8XUSinemdROQfInLI+bdjQxxofT0wpjfBQcLv11tfhTGmZaszKEQkGFgEfBvoD0wXkf7VLBcJLAC2VE5T1eWqmqKqKcBMIFtVd4pICPA7YLyqDgZ2A/Oc1X4BrFHVZGCN877JxbaLYPrwON7ankPOGWtVGGNaLl9aFGlApqpmqWoJnrGvJ1ez3ELgKaC4hu1MB1Y4r8X5aSOeoeXaASeceZOBZc7rZcA/+1Bjo3hgbCIi8MIG66swxrRcvgRFd+CY1/scZ1oVERkCxKnq6lq2MxUnKFS1FHgI+BxPQPQHXnaW66yqJ53lTgKxPtTYKLp1aMWUYXGs3JbDl4U15Z8xxgQ2X4KiusGkqwbaFpEg4Gng0Ro3IDICKFLVPc77UDxBMQTohufU0y99LxtEZK6IpItIel5eXn1WrZeHxyVSoWqtCmNMi+VLUOQAcV7ve/DVaSKASGAgsF5EsoGRwKrKDm3HNL467QSQAqCqh1VVgZXAKGfeKRHpCuD8m1tdUaq6WFVTVTU1JibGh8O4OnGdWnPHkO6s2HqU3PPWqjDGtDy+BMU2IFlEEkQkDM+X/qrKmapaqKrRqhqvqvHAZmCSqqZDVYvjLjx9G5WOA/1FpPIb/lZgv/N6FTDLeT0LePeqjqwBPTI+idLyCl78KMvtUowxpsnVGRSqWobniqT38XyZr1TVvSLyhIhM8mEfY4AcVa36llXVE8C/AR+JyG48LYz/58x+ErhVRA7hCZAn63NAjSE+ug2TU7rzx81HKbhw2e1yjDGmSYnnzE/zlpqaqunp6Y26j8zcC9z69AYeHJvIzyf2a9R9GWNMUxCR7aqaWtdydme2j5Ji23L7oK68ujGbMxdL3C7HGGOajAVFPcyfkMzFknKWfnrE7VKMMabJWFDUQ98ukUwc0IWln2ZTeKnU7XKMMaZJWFDU0/ybkzh/uYxlG7PdLsUYY5qEBUU9DejWnluui+XlT45wvthaFcaYwGdBcRXmT0im8FIpr23+wu1SjDGm0VlQXIXr4zowtk8ML318hKKSMrfLMcaYRmVBcZUW3JzM6YslLN981O1SjDGmUVlQXKVhvToyOimKP3yURXFpudvlGGNMo7GguAYLJiSTf+EyK7Zaq8IYE7gsKK7BiN5RpCV04oUNh61VYYwJWBYU1+hfbk7m1LnLvLk9x+1SjDGmUVhQXKNRiVEM7dmB59dlUlJW4XY5xhjT4CworpGIsODmZE4UFvPODmtVGGMCjwVFAxjbJ4bBPdqzaH0mpeXWqjDGBBafgkJEJopIhohkisgvalluioho5TCoIjJDRHZ6/VSISIqIRF4xPV9EfuusM1tE8rzm3dcwh9p4RIQFE5I5dvoS7+48UfcKxhjTjITUtYCIBAOL8Iw2lwNsE5FVqrrviuUigQXAlsppqrocWO7MHwS8q6o7ndkpXutuB97x2twbqjrvqo7IJTdfF0v/ru1YtC6TO4Z0JzhI3C7JGGMahC8tijQgU1WzVLUEz9jXk6tZbiHwFFBcw3amAyuunCgiyUAs8LFPFfspT19FEkfyL7J6t7UqjDGBw5eg6A4c83qf40yrIiJDgDhVXV3LdqZSTVDgCZA39Otjst4pIrtF5C0RifOhRr/wrf5d6Ns5kmfXZlJR0fyHmDXGGPAtKKo7h1L1LSgiQcDTwKM1bkBkBFCkqnuqmT2NrwfIe0C8qg4GPgSW1bDNuSKSLiLpeXl5dR9FEwgKEuZNSCIz9wJ/2/Ol2+UYY0yD8CUocgDvv+p7AN7nViKBgcB6EckGRgKrKju0HVeGAQAicj0QoqrbK6epaoGqXnbevggMq64oVV2sqqmqmhoTE+PDYTSN7wzqSu+YNjy79pC1KowxAcGXoNgGJItIgoiE4fnSX1U5U1ULVTVaVeNVNR7YDExS1XSoanHchadv40rf6LcQka5ebycB++txPK4LDhLmT0jiwJfn+cf+U26XY4wx16zOoFDVMmAe8D6eL+2VqrpXRJ4QkUk+7GMMkKOqWdXM+z7fbGksEJG9IrILz1VUs33Yh1/57uBu9IpqzbNrD/H1rhdjjGl+JBC+yFJTUzU9Pd3tMr5mZfoxfvbWbpbMTmVCv85ul2OMMd8gIttVNbWu5ezO7EZyx5Du9OjYimfWZFqrwhjTrFlQNJLQ4CAeHpfEzmNn+fhQvtvlGGPMVbOgaER3DutO1/YRPLPG+iqMMc2XBUUjCg8J5sGxiaR/cYZNWQVul2OMMVfFgqKRTR0eR2xkOM+uyXS7FGOMuSoWFI0sIjSYuWN6symrgG3Zp90uxxhj6s2CognMGNGL6LZhPLPmkNulGGNMvVlQNIFWYcHcd1NvPj6Uz2dHz7hdjjHG1IsFRROZObIXHVuH8uxa66swxjQvFhRNpE14CPfemMDaA7l8nlPodjnGGOMzC4omdPeoeNpFhPDsWuurMMY0HxYUTahdRCj3jE7gg32n2H/ynNvlGGOMTywomtic0Qm0DQ/hOeurMMY0ExYUTax961BmjerFX/ec5NCp826XY4wxdbKgcMG9N/amVWgwz62zVoUxxv/5FBQiMlFEMkQkU0R+UctyU0REK4dBFZEZIrLT66dCRFJEJPKK6fki8ltnnXARecPZ1xYRiW+IA/UnndqEMXNkL97bdYKsvAtul2OMMbWqMyhEJBhYBHwb6A9MF5H+1SwXiWdEui2V01R1uaqmqGoKMBPIVtWdqnq+croz7wvgHWe1e4EzqpoEPA38+7Udon+676behIUEsWjdYbdLMcaYWvnSokgDMlU1S1VL8Ix9Pbma5RYCTwHFNWznG+NjA4hIMhALfOxMmgwsc16/BdwsIuJDnc1KTGQ4P0jrxX/vPM7RgiK3yzHGmBr5EhTdgWNe73OcaVVEZAgQp6qra9nOVKoJCjwB8oZ+NWBD1f6c8boLgSgf6mx2Hhjbm+Ag4ffrra/CGOO/fAmK6v6arxqFR0SC8JwierTGDYiMAIpUdU81s6fx9QCpdX9e25wrIukikp6Xl1fTrv1a53YRTBsex1vbc8g5Y60KY4x/8iUocoA4r/c9gBNe7yOBgcB6EckGRgKrKju0HVeGAQAicj0Qoqrbq9ufiIQA7YFvPJ9bVReraqqqpsbExPhwGP7pwbGJiMALG6yvwhjjn3wJim1AsogkiEgYni/9VZUzVbVQVaNVNV5V44HNwCRVTYeqFsddePo2rlRdv8UqYJbzegqwVgN4HNFuHVoxZVgcK7fl8GVhTd07xhjjnjqDwuknmAe8D+wHVqrqXhF5QkQm+bCPMUCOqmZVM+/7fDMoXgaiRCQT+DFQ4+W4geLhcYlUqFqrwhjjlyQQ/lhPTU3V9PR0t8u4Jj99cxerdp3g45+PJzYywu1yjDEtgIhsV9XUupazO7P9xCPjkygtr+DFj6preBljjHssKPxEfHQbJqd054+bj1Jw4bLb5RhjTBULCj/yyPgkisvKeemTI26XYowxVSwo/EhSbFtuH9SVVzdmc7aoxO1yjDEGsKDwO/MnJHOxpJwl1qowxvgJCwo/07dLJBMHdGHpxmwKL5W6XY4xxlhQ+KP5NydxvriMZRuz3S7FGGMsKPzRgG7tueW6WF7+5AgXLpe5XY4xpoWzoPBT8yckU3iplFc3ZbtdijGmhbOg8FPXx3VgbJ8YXvr4CEUl1qowxrjHgsKPLbg5mdMXS1i++ajbpRhjWjALCj82rFdHRidF8YePsiguLXe7HGNMC2VB4ecWTEgm/8JlVmy1VoUxxh0WFH5uRO8o0hI68cKGw9aqMMa4woKiGVgwIZlT5y7z5vYct0sxxrRAFhTNwOikKIb27MAL6w9TUlbhdjnGmBbGp6AQkYkikiEimSJS44hzIjJFRLRyvGwRmSEiO71+KkQkxZkXJiKLReSgiBwQkTud6bNFJM9rnfsa4kCbMxFh/s3JHD97iXd2WKvCGNO06gwKEQkGFgHfBvoD00WkfzXLRQILgC2V01R1uaqmqGoKMBPIVtWdzuzHgFxV7eNsd4PX5t6oXE9VX7rKYwso4/rEMLhHexatz6S03FoVxpim40uLIg3IVNUsVS0BXgcmV7PcQuApoLiG7Uzn6+NjzwF+DaCqFaqa73PVLZCIMH9CMsdOX+LdnSfcLscY04L4EhTdgWNe73OcaVVEZAgQp6qra9nOVJygEJEOzrSFIrJDRN4Ukc5ey94pIrtF5C0RiatuYyIyV0TSRSQ9Ly/Ph8No/m65LpbrurZj0bpMyiua/1jnxpjmwZegkGqmVX1LiUgQ8DTwaI0bEBkBFKnqHmdSCNAD+FRVhwKbgN84894D4lV1MPAhsKy6barqYlVNVdXUmJgYHw6j+RMRFkxI4kj+RVbvtlaFMaZp+BIUOYD3X/U9AO9vqUhgILBeRLKBkcCqyg5txzS+ftqpACgC/uy8fxMYCqCqBapaOWj0i8Awn46khbhtQBf6dG7Ls2szqbBWhTGmCfgSFNuAZBFJEJEwPF/6qypnqmqhqkararyqxgObgUmqmg5VLY678PRtVK6jeFoO45xJNwP7nOW7eu17ErD/6g4tMAUFCfMmJJOZe4G/7fnS7XKMMS1AnUGhqmXAPOB9PF/aK1V1r4g8ISKTfNjHGCBHVbOumP5z4HER2Y3niqjKU1cLRGSviOzCcxXVbN8OpeW4fVBXese04dm1h6xVYYxpdOL54755S01N1fT0dLfLaFLv7Mjhxyt38YeZw7htQBe3yzHGNEMisl1VU+tazu7MbqYmXd+NXlGteXbtIQIh7I0x/suCopkKCQ7ikXFJ7Dl+jnUZuW6XY4wJYBYUzdgdQ7vTvUMrnlmTaa0KY0yjsaBoxkKDg3h4fCI7j53lk0y7sd0Y0zgsKJq5KcN60LV9BM+ssb4KY0zjsKBo5sJDgnlwbCLbss+wOeu02+UYYwKQBUUAmDo8jtjIcJ5Zc8jtUowxAciCIgBEhAYzd0xvNmUVsC3bWhXGmIZlQREgZozoRXTbMGtVGGManAVFgGgVFsx9N/Xm40P5fHb0jNvlGGMCiAVFAJk5shcdW4fy7NpMt0sxxgQQC4oA0iY8hHtvTGDtgVz2HC90uxxjTICwoAgwd4+Kp11EiPVVGGMajAVFgGkXEco9oxP4YN8p9p8853Y5xpgAYEERgOaMTqBteAjPWV+FMQFtfUYupeUVjb4fn4JCRCaKSIaIZIrIL2pZboqIaOUwqCIyQ0R2ev1UiEiKMy9MRBaLyEEROSAidzrTw0XkDWdfW0Qk/toPs2Vp3zqUWaN68dc9Jzl06rzb5RhjGsGqXSeYvXQbL3585ZhwDa/OoBCRYGAR8G2gPzBdRPpXs1wknhHptlROU9Xlqpqiqil4RrHLVtWdzuzHgFxV7eNsd4Mz/V7gjKomAU8D/361B9eS3Xtjb1qFBvPcOmtVGBNoNh7O5ycrd5EW34k5oxMafX++tCjSgExVzVLVEjxjX0+uZrmFwFNAcQ3bmQ6s8Ho/B/g1gKpWqGrl408nA8uc128BN4uI+FCn8dKpTRgzR/bivV0nyMq74HY5xpgGsv/kOR54dTvx0a158e5UIkKDG32fvgRFd+CY1/scZ1oVERkCxKnq6lq2MxUnKESkgzNtoYjsEJE3RaTzlftzxusuBKJ8qNNc4b6behMWEsSidYfdLsUY0wCOn73E7KVbaRMewiv3pNG+dWiT7NeXoKjur/mq51mLSBCeU0SP1rgBkRFAkarucSaFAD2AT1V1KLAJ+I0v+/Pa5lwRSReR9Ly8PB8Oo+WJiQznB2m9+O+dxzlaUOR2OcaYa3C2qIRZS7ZSVFLOK3OG061Dqybbty9BkQPEeb3vAZzweh8JDATWi0g2MBJYVdmh7ZjG1087FQBFwJ+d928CQ6/cn4iEAO2BbzzpTlUXq2qqqqbGxMT4cBgt0wNjexMcJPx+vfVVGNNcFZeWc/+r6RwtKGLxzFT6dWnXpPv3JSi2AckikiAiYXi+9FdVzlTVQlWNVtV4VY0HNgOTVDUdqlocd+Hp26hcR4H3gHHOpJuBfc7rVcAs5/UUYK3aiDxXrXO7CKYNj+PtHTnknLFWhTHNTXmF8qM3drIt+wz/NfV6bkhs+jPxdQaF008wD3gf2A+sVNW9IvKEiEzyYR9jgBxVvfIarp8Dj4vIbjxXRFWeunoZiBKRTODHQI2X4xrfPDg2EYAXNlhfhTHNiaryxHt7+dueL/nXf+rPPw3u5kodEgh/rKempmp6errbZfi1X77zOW9vz+Gjn42nS/sIt8sxxvjghQ2HefJvB7j/pgQeu/0bdyVcMxHZrqqpdS1nd2a3EA+PS6Rc1VoVxjQTf/4shyf/doDvXt+NX377OldrsaBoIeI6teZ7Q7qzYutRcs/XdKuLMcYffHwoj5++uZsbekfxm7sGExTk7q1kFhQtyCPjkygtr+DFjxr/ln9jzNXZc7yQB1/bTlJsW/5w9zDCQxr/hrq6WFC0IPHRbZic0p0/bj5KwYXLbpdjjLnCsdNF3PPKNjq0DuOVe9JoF9E0N9TVxYKihXlkfBLFZeW89MkRt0sxxng5fdFzQ11JWQXL5gz3q4tOLChamKTYttw+qCuvbszmbFGJ2+UYY4BLJeXct2wbOWcv8dKsVJJiI90u6WssKFqgeROSuFhSzhJrVRjjurLyCuav+IzPjp3lmWkpDI/v5HZJ32BB0QL169KOiQO6sHRjNoWXSt0ux5gWS1X5X6v28uH+Uzz+3QFMHNjV7ZKqZUHRQs2bkMT54jKWbcx2uxRjWqxF6zL505ajPDQukVmj4t0up0YWFC3UwO7tueW6WJZ8eoQLl8vcLseYFufN9GP85oODfG9Id352W1+3y6mVBUULNn9CMmeLSnlt0xdul2JMi7IuI5dfvPM5NyVH8+Sdg/H3sdksKFqw6+M6MLZPDC9+nEVRibUqjGkKu46d5ZHlO+jXJZLnfziMsBD//xr2/wpNo1pwcxKnL5bwpy1H3S7FmID3RcFF5ryyjU5twlh6z3Dahoe4XZJPLChauGG9OjE6KYoXNmRRXFrudjnGBKz8C5e5e8lWKlRZNieN2Ej/uaGuLhYUhvkTksm/cJnXt1qrwpjGUFRSxr2vbOPUuWJenj2cxJi2bpdULz4FhYhMFJEMEckUkRoHEhKRKSKilcOgisgMEdnp9VMhIinOvPXONivnxTrTZ4tIntf0+xriQE3NRvaOIi2hEy9syOJymbUqjGlIZeUVPLJ8B58fL+TZ6UMZ2rOj2yXVW51BISLBwCLg20B/YLqIfGMEDRGJBBYAWyqnqepyVU1R1RQ8o9hlq+pOr9VmVM5X1Vyv6W94TX/p6g7N1MeCCcl8ea6YN9Nz3C7FmIChqjz25z2sy8hj4T8P5Nb+nd0u6ar40qJIAzJVNUtVS/CMfT25muUWAk8BNQ12MB1YcVVVmkY3OimKoT078Pz6w5SUVbhdjjEB4bcfHuKN9GMsmJDEjBG93C7nqvkSFN2BY17vc5xpVURkCBCnqqtr2c5UvhkUS53TS/8qX7+Q+E4R2S0ib4lInA81mmskIsy/OZnjZy/x58+sVWHMtVqx9Si/W3OIu4b14Ee39nG7nGviS1BUdydI1UDbIhIEPA08WuMGREYARaq6x2vyDFUdBNzk/Mx0pr8HxKvqYOBDYFkN25wrIukikp6Xl+fDYZi6jOsTw+Ae7Vm07jBl5daqMOZqrdl/isf+/Dnj+sbw/743yO9vqKuLL0GRA3j/Vd8DOOH1PhIYCKwXkWxgJLCqskPbMY0rWhOqetz59zzwJzynuFDVAlWtHFXnRWBYdUWp6mJVTVXV1JiYGB8Ow9RFRJg/IZmjp4t4d+eJulcwxnzDZ0fP8MifdjCwe3sW/WAoocHN/+JSX45gG5AsIgkiEobnS39V5UxVLVTVaFWNV9V4YDMwSVXToarFcReevg2caSEiEu28DgX+CdjjvPd+fOIkYP81HJ+pp1uui+W6ru1YtC6T8gqtewVjTJWsvAvMeWUbndtFsGT2cNo0kxvq6lJnUKhqGTAPeB/Pl/ZKVd0rIk+IyCQf9jEGyFFV74Gaw4H3RWQ3sBM4jqf1ALBARPaKyC48V1HN9vlozDUTERZMSCIr/yKrd1urwhhf5Z4vZtbSrQSJsOyeNKLbhrtdUoMR1eb/V2Nqaqqmp6e7XUbAqKhQJv7uI1Th/f8xhqCg5n1+1ZjGduFyGdMWb+Jw7kVenzuS6+M6uF2ST0Rku6qm1rVc8z95ZhpcUJAwb0Iyh3Iv8Pe9X7pdjjF+rbS8gof+uJ39J8/z+xlDm01I1IcFhanW7YO60jumDc+sOUSF9VUYUy1V5edv7+bjQ/n8+o5BjO8X63ZJjcKCwlQrOEiYNz6JA1+e58P9p9wuxxi/9JsPMnhnx3F+fGsfvj88cG/5sqAwNZp0fTd6RbXmmbWHCIS+LGMa0mubslm07jDT03oyf0KS2+U0KgsKU6OQ4CAeGZfEnuPnWJ9hNzUaU+nve77kf63ayy3XxbJw8oBmf0NdXSwoTK3uGNqd7h1a8bs11qowBiA9+zT/8vpnXN+jA89OH0pIANxQV5fAP0JzTUKDg3h4fCI7j53lk8x8t8sxxlWZuee5d1k63Tq0Ysns4bQKC3a7pCZhQWHqNGVYD7q2j+AZa1WYFuzUuWJmLdlGaHAQy+5Jo1ObMLdLajIWFKZO4SHBPDg2kW3ZZ9icddrtcoxpcueKS5m1ZCtni0p45Z7h9Ixq7XZJTcqCwvhk6vA4YiPD+e2HB7lUYqPgmZajpKyCB1/bTmbuBZ7/4TAGdm/vdklNzoLC+CQiNJiHxyWy5chprv+3D5i+eDOL1mWy89hZe3igCVgVFcpP39rFxsMFPDVlMGP6tMwnVQfGow1Nk5g1Kp7eMW35+FAen2QW8B/vZ/Af72fQLiKEGxKjuDEpmtFJ0SREtwn4ywVNy/Dvfz/AuztP8LOJffne0B5ul+MaCwrjMxFhTJ+Yqr+q8i9cZuPhAj49lM8nmfm8v9dzB3e39hGMTormxuRoRiVGExMZOE/RNC3Hkk+O8IePsrj7hl48NDbR7XJcZU+PNQ1CVfmioIhPMvP5NDOfjYcLKLxUCkC/LpGe4EiKJi2hU8A8o98Err/sPsm8FTv4Vv/O/H7GMIID9AnKvj491oLCNIryCmXvicKq4NiWfYaSsgpCgoShPTs6LY4oBvfoEBAjgJnAsTmrgLtf3srgHu35430jiAgN3HslGjQoRGQi8DsgGHhJVZ+sYbkpwJvAcFVNF5EZwE+9FhkMDFXVnSKyHugKXHLmfUtVc0UkHHgVzxCoBcBUVc2urT4LCv9XXFpOevaZquDYc6IQVWgbHsLI3p0YnRTNTcnRJMa0tf4N45qML88z5YWNxEaG8/ZDo+jQOrDvlfA1KOo8ByAiwcAi4FY842diRM8+AAARN0lEQVRvE5FVqrrviuUi8YxIt6VymqouB5Y78wcB76rqTq/VZlQOmerlXuCMqiaJyDTg34GpddVp/FtEaDA3Jnv6LQDOXCxhU1ZBVXB8uD8XgM7twqtOU41OiqZzuwg3yzYtyImzl5i9dCutQoNZNict4EOiPnw5WZwGZFYOZSoirwOTgX1XLLcQeAr4SQ3bmQ6s8GF/k4HHnddvAc+JiGggnCMzVTq2CeM7g7rynUGeIdKPnS7i00xPp/j6jDze2XEcgOTYtlXBMaJ3JyIjQt0s2wSowkulzF66lfPFZax84AZ6dGxZN9TVxZeg6A4c83qfA4zwXkBEhgBxqrpaRGoKiql4QsDbUhEpB94G/o8TBlX7U9UyESkEogB70FAAi+vUmmlpPZmW1pOKCmX/l+ec4Cjg9W1HeWVjNsFBQkpch6rgSInrQFiI9W+Ya1NcWs7cV9M5kn+RZfek0b9bO7dL8ju+BEV1J4yr/roXkSDgaWB2jRsQGQEUqeoer8kzVPW4c8rqbWAmnr6JWvfntc25wFyAnj171n0UptkIChIGdGvPgG7tmTsmkctl5ez44mxVi+O5tYd4Zs0hWocFMyKhU9WluH07R1r/hqmXigrl0ZW72HLkNL+blsKopGi3S/JLvgRFDuA9dFMP4ITX+0hgILDe+Z+0C7BKRCZ59T9M44rTTqp63Pn3vIj8Cc8prle99pcjIiFAe+AbDxhS1cXAYvB0ZvtwHKaZCg8J5obEKG5IjOInt/Wl8FIpm7MKqoJj3V/2AxDdNpzRSVGMdvo3undo5XLlxp+pKgv/so+/fH6Sx75zHZNTurtdkt/yJSi2AckikgAcx/Ol/4PKmapaCFTFsHM1008qQ8JpcdwFjPFaJgTooKr5IhIK/BPwoTN7FTAL2ARMAdZa/4Tx1r5VKLcN6MJtA7oAnk7IT51O8U8yC3h3p+fvmN7RbapC44beUbRvbf0b5isvfXyEpZ9mM2d0AvfdlOB2OX6tzqBw+gnmAe/juTx2iaruFZEngHRVXVXHJsYAOZWd4Y5w4H0nJILxhMSLzryXgddEJBNPS2JavY7ItDjdOrTirtQ47kqNQ1U5eOpC1dVUb+/I4bXNXxAkMKhHB250WhzDenUkPCRwr483tXt353H+71/3c/vgrvzP26+zU5Z1sBvuTEArKatgV85ZPnEeM1L5EMOI0CCGx3equgy3f9d2BAXo3bfm6zZm5jNr6VaG9uzIsjlpAX1DXV3szmxjqnG+uJQtWaf5JDOfjYfzOXjqAgAdW4cyyrma6sakaOI62eWRgWjfiXNM/cMmunaI4M0HR9G+Vcs+HdlgN9wZE0giI0K5pX9nbunfGfCMWlbZKf5pZj5/2X0SgJ6dWlddhjsqMYqOLWg0s0CVc6aI2Uu30jYihGVz0lp8SNSHtSiMcagqh/MuOKepCticVcCFy2WIwIBu7aqCY3h8pxZ9uqI5OltUwp3PbyT3/GXeenAUfbtEul2SX7BTT8Zco7LyCnblFFa1OD47eobSciUsJIjUXh2rgmNg9/YB+3TRQFBcWs4PX9rC7pxCXr03jZG9o9wuyW9YUBjTwC5eLmNr9umq8TcOfHkegHYRIYxK9Nz0d2NSNL2iWttVNH6ivEJ5ePl2Pth3iuemD+X2wV3dLsmvWB+FMQ2sTXgI4/vGMr5vLAB55y+z8bBz/8ahfP6+90sAundo5bmaKtnTvxHd1gZucoOq8viqvby/9xT/+7v9LSSugQWFMVcpJjKcySndmZzSHVUlu3LgpkP5/G3PSd5I9zwibXCP9p6A6RfL4O7t7TLcJvL8hsO8tvkLHhjTm3tG2w1118JOPRnTCMorlD3HC/noYB7rMnL57NhZVCG6bRhj+8QyoV8sN/WJpp09DbdRvL09h0ff3MXklG48/f0UC+caWB+FMX7k9MUSPjqYx9oDuWw4mEfhpVJCgoRhvToyoZ8nOJJibdCmhvDRwTzmvLKNEb07sXR2mj1huBYWFMb4qbLyCj47dpZ1B3JZeyC3qlO8R8dWjO/rCY0bEqPsEtyrsOd4IVP/sImeUW1Y+cBIG7+kDhYUxjQTJ85eYl1GLusO5PJpZgGXSsuJCA1iVGI04/vGML5frA2k44Njp4u44/cbCQ8J4p2HR9noiD6woDCmGSouLWfLkdNVrY2jp4sA6NO5LeP7xTKhbyxDe3UkNNhOp3g7fdFzQ93piyW8/dANJMXaDXW+sKAwpplTVbLyL1aFxtYjpymrUCIjQhjTJ4YJfWMZ1zeGqBZ++e2lknKmv7iZ/SfPsfy+EaTGd3K7pGbD7qMwppkTERJj2pIY05b7burN+eJSPs3MZ+2BXNZl5PGX3ScRgcE9OjDB6dsY0K1lPQW3rLyC+St2sDvnLL+fMcxCopFYi8KYZqiiQtl74pwTGrnsyvFcfhsTGc64PjFM6BfLjcnRAd2Zq6r86s+fs2LrMRb+80BmjuzldknNjp16MqYFyb9wmQ0ZeazNyOWjg3mcLy4jNFgYHt+JCf1iGdc3lsSYNgF1+e0zaw7xX/84yCPjE/npbf3cLqdZatCgEJGJwO/wjEb3kqo+WcNyU4A3geGqmi4iM4Cfei0yGBiqqju91lkF9FbVgc77x4H7gTxnkV+p6l9rq8+CwpivlJVXsP2LM6x1rqSqHHOjZ6fWTOjnuUN8RELzfgLuym3H+Nnbu/ne0O78513XB1QANqUGCwoRCQYOArcCOXjG0J6uqvuuWC4S+AsQBsyrHDPba/4g4F1V7e017Xt4xsUefEVQXFDV39RVfCULCmNqlnOmiHUZeaw7kMvGw/kUl1bQKjSY0UlRjO/neXZVtw6t3C7TZ+sO5HLfq+mMTorm5VmpdgXYNWjIzuw0ILNyzGsReR2YDOy7YrmFwFPAT2rYznRghVeBbYEfA3OBlT7UYYy5Cj06tmbmyF7MHNmL4tJyNh0uYF2G50qqD/fnAtCvS6Tn8tt+sQyJ60CIn3757jp2loeX7+C6rpH8fsZQC4km4ktQdAeOeb3PAUZ4LyAiQ4A4VV0tIjUFxVQ8AVNpIfCfQFE1y84TkbuBdOBRVT1z5QIiMhdPyNCzZ08fDsMYExEa7GlF9Ivl3yYpmbkXWOtcfrv4oyyeX3+Y9q1CGdsnhvH9YhjbJ5ZOfjK6X3b+Rea8so3oyDCWzB5O23C7aLOp+PJJV3fyr+p8lYgEAU8Ds2vcgMgIoEhV9zjvU4AkVf2RiMRfsfjzeEJE+SpM5nyjANXFwGLwnHry4TiMMV5EhOTOkSR3juSBsYkUXirlk0P5zvOoclm16wQiMCSuQ1WH+IBu7VzpD8i/cJlZS7dSocqye9KIjbS7rpuSL0GRA8R5ve8BnPB6HwkMBNY7v0BdgFUiMsmrn2IaXqedgBuAYSKS7dQQKyLrVXWcqp6qXEhEXgRW1++QjDFXo32rUG4f3JXbB3elokL5/Hhh1eW3v/ngIL/54CCd24VXPTJ9dFJ0k/xVf/FyGXNe2capc8X86f6R9I5p2+j7NF/nS2d2CJ7O7JuB43g6s3+gqntrWH498JPKkHBaHEeBMZX9HFcsHw+s9urM7qqqJ53XPwJGqOq02mq0zmxjGlfu+WI2ZHgemf7xwXzOXy4jLDiItIROVX0bCdFtGny/peUV3P9qOh8dzGPxzFRu6d+5wffRkjVYZ7aqlonIPOB9PJfHLlHVvSLyBJCuqqvq2MQYIKe6kKjBU86pKQWygQd8XM8Y00hiIyO4KzWOu1LjKC2vYFu253lU6zLyWLh6HwtX7yMhug3j+npu9ktL6ER4yLVdfquq/Oqdz1mfkcevvzfIQsJFdsOdMeaaHC0oqrqKalNWASVlFbQOC+bGpOiqy2+7tK9/n8J/fZDBM2sz+Zebk/nRrX0aoXJjd2YbY5rcpZJyNh52nkd1IJcThcUA9O/arupmv5S4DgTX8Tyq5Vu+4LE/72FqahxP3jnIbqhrJBYUxhhXqSoZp86z7oDnZr/tR89QXqF0bF15+W0sY/vE0KH11y+//ce+UzzwWjpj+8Tw4t2pfntPRyCwoDDG+JXColI2HMpj/YFc1h/M4/TFEoIEhvbsWNUhXlRSzoyXNtO3cyQr5o6kdZjdK9GYLCiMMX6rvELZlfPVcLB7T5wDQAR6dWrNWw+NIrqFj7PRFGw8CmOM3woOEob27MjQnh159Ft9OXWumPUZuXx29CwPj0uykPAzFhTGGNd1bhfB1OE9mTrcHsfjj6yXyBhjTK0sKIwxxtTKgsIYY0ytLCiMMcbUyoLCGGNMrSwojDHG1MqCwhhjTK0sKIwxxtQqIB7hISJ5wBdXuXo0kN+A5TQUq6t+rK7689farK76uZa6eqlqTF0LBURQXAsRSfflWSdNzeqqH6ur/vy1NqurfpqiLjv1ZIwxplYWFMYYY2plQQGL3S6gBlZX/Vhd9eevtVld9dPodbX4PgpjjDG1sxaFMcaYWgV0UIjIRBHJEJFMEflFNfNni0ieiOx0fu7zmjdLRA45P7P8qK5yr+mrmrIuZ5nvi8g+EdkrIn/ymu7a51VHXa59XiLytNe+D4rIWa95bv5+1VaXm59XTxFZJyKfichuEfmO17xfOutliMht/lCXiMSLyCWvz+uFJq6rl4iscWpaLyI9vOY17O+XqgbkDxAMHAZ6A2HALqD/FcvMBp6rZt1OQJbzb0fndUe363LmXXDx80oGPqv8LIBYP/m8qq3L7c/riuXnA0v84fOqqS63Py8859ofcl73B7K9Xu8CwoEEZzvBflBXPLDHxc/rTWCW83oC8Fpj/X4FcosiDchU1SxVLQFeByb7uO5twD9U9bSqngH+AUz0g7oaky913Q8scj4TVDXXme7251VTXY2pvv8dpwMrnNduf1411dWYfKlLgXbO6/bACef1ZOB1Vb2sqkeATGd7btfVmHypqz+wxnm9zmt+g/9+BXJQdAeOeb3PcaZd6U6n6faWiMTVc92mrgsgQkTSRWSziPxzA9Xka119gD4i8qmz/4n1WNeNusDdzwvwnCLA85fw2vqu28R1gbuf1+PAD0UkB/grntaOr+u6URdAgnNKaoOI3NRANfla1y7gTuf1HUCkiET5uG69BHJQSDXTrrzE6z0gXlUHAx8Cy+qxrht1AfRUz12YPwB+KyKJTVhXCJ7TPOPw/CX6koh08HFdN+oCdz+vStOAt1S1/CrWra9rqQvc/bymA6+oag/gO8BrIhLk47pu1HUSz+c1BPgx8CcRaUfD8KWunwBjReQzYCxwHCjzcd16CeSgyAG8/xLvwRVNRlUtUNXLztsXgWG+rutSXajqCeffLGA9MKSp6nKWeVdVS51TABl4vqBd/bxqqcvtz6vSNL5+esftz6umutz+vO4FVjr73wRE4HmOkdufV7V1OafCCpzp2/H0KfRpqrpU9YSqfs8JqsecaYU+HlP9NEZHjD/84PkrMwtP07qyM2jAFct09Xp9B7BZv+oMOoKnI6ij87qTH9TVEQh3XkcDh6ilo7IR6poILPPa/zEgyg8+r5rqcvXzcpbrC2Tj3LPkD79ftdTl9u/X34DZzuvr8Hy5CTCAr3dmZ9FwndnXUldMZR14Op2PN/HvfTQQ5Lz+v8ATjfX7dc0H5M8/eJqJB/Ek/WPOtCeASc7rXwN7nf8I64B+XuvOwdNplgnc4w91AaOAz53pnwP3NnFdAvwXsM/Z/zQ/+byqrcvtz8t5/zjwZDXruvZ51VSX258Xns7ZT5397wS+5bXuY856GcC3/aEuPP0Dlf+f7gC+28R1TcET5geBl3BCvjF+v+zObGOMMbUK5D4KY4wxDcCCwhhjTK0sKIwxxtTKgsIYY0ytLCiMMcbUyoLCGGNMrSwojDHG1MqCwhhjTK3+Pw5dO6X9Me+pAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "train_means = grid_search.cv_results_['mean_train_score']\n",
    "train_stds = grid_search.cv_results_['std_train_score']\n",
    "\n",
    "x_axis = sub_sample_s\n",
    "plt.plot(x_axis, -test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 每棵树的列采样比例colsample_bytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:16:11.172155Z",
     "start_time": "2020-06-25T08:00:20.891419Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed: 12.7min remaining:  3.2min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 15.8min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=1.0, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=40,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=380,\n",
       "        n_jobs=-1, num_class=9, num_leaves=60, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.8, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score=True,\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "         'objective': 'multiclass',\n",
    "         'num_class': 9,\n",
    "         'learning_rate': 0.1,\n",
    "         'n_estimators': n_estimators_1,\n",
    "         'max_depth': 7,\n",
    "         'num_leaves': 60,\n",
    "         'min_child_samples': 40,\n",
    "         'max_bin': 127,\n",
    "         'subsample': 0.8,\n",
    "         'bagging_freq': 1,\n",
    "         }\n",
    "lg = LGBMClassifier(silent = False, **params)\n",
    "\n",
    "colsample_bytree_s = [i / 10.0 for i in range(5, 10)]\n",
    "tuned_parameters = dict(colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg,\n",
    "                           n_jobs = 4,\n",
    "                           param_grid = tuned_parameters,\n",
    "                           cv = kfold,\n",
    "                           scoring = 'neg_log_loss', \n",
    "                           verbose = 5,\n",
    "                           refit = False,\n",
    "                           return_train_score = True)\n",
    "grid_search.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:16:24.944800Z",
     "start_time": "2020-06-25T08:16:24.940838Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.473247773421\n",
      "{'colsample_bytree': 0.5}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:16:28.664527Z",
     "start_time": "2020-06-25T08:16:28.595635Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8FeXZ//HPRdhENtkJi6HIvgVJQVyoS6u4gQsqVK0KVm2l2Met9rF92qq1LbZV28faYoviUlDozxopaq1KVRQlaNgXEaGEsASQJUKAkOv3x5nwHGJycgJJ5pyT7/v1Oi/OzNwz55oxnuvcc8/MZe6OiIhIReqFHYCIiCQ2JQoREYlJiUJERGJSohARkZiUKEREJCYlChERiUmJQkREYlKiEBGRmJQoREQkpvphB1Ad2rRp4xkZGWGHISKSVBYuXLjN3dtW1i4lEkVGRgY5OTlhhyEiklTMbH087XTqSUREYlKiEBGRmJQoREQkJiUKERGJSYlCRERiUqIQEZGYlChERCQmJQoRkSR0oLiEB+esYNOufTX+WUoUIiJJZtfeg1w39UOmvL2WN1ZsrfHPS4k7s0VE6op1275g/FMLyPt8H7+9chCXndy5xj9TiUJEJEl8sHY7Nz+7EAOevXEYQ7u1qpXPjevUk5mNNLNVZrbGzO6J0W6MmbmZZQXTV5tZbtSrxMwyg2UNzWyKma02s5Vmdnkwv5GZPR981gdmlnHsuykiktxmLczjmr98QOvjG/L3W0+rtSQBcfQozCwNeAz4BpAHLDCzbHdfXqZdM2AS8EHpPHd/DnguWD4AeMndc4PF9wJb3b2nmdUDSvd6AvC5u59kZmOBXwFXHcM+iogkrZIS59f/XMUf5n7Kqd1b8/jVQ2jRpEGtxhBPj2IosMbd17r7AWAGMLqcdvcDk4GiCrYzDpgeNT0e+AWAu5e4+7Zg/mhgWvB+FnCOmVkccYqIpJR9Bw4xcfpH/GHup4wb2oVp44fWepKA+BJFJ2BD1HReMO8wMxsMdHH32TG2cxVBojCzlsG8+83sIzObaWbty36euxcDu4DWccQpIpIytu4pYuyU93ll6WbuvaAPD146gAZp4VyoGs+nlvdr3g8vjJw2ehi4o8INmA0D9rr70mBWfaAzMM/dTwbeB34dz+dFbfMmM8sxs5yCgoI4dkNEJDms2LSbS/53Hqu3FPKna4bw7RFfIcwTK/EkijygS9R0ZyA/aroZ0B+Ya2brgFOA7NIB7cBYjjzttB3YC7wYTM8ETi77eWZWH2gB7CgblLtPcfcsd89q27bSAk0iIknhzZVbGPP4e5Q4zLxlOOf26xB2SHEligVADzPrZmYNiXzpZ5cudPdd7t7G3TPcPQOYD4xy9xw43OO4gsjYRuk6DrwMnBnMOgcoHRzPBq4L3o8B3gzai4ikLHfnyXmfceO0HDLaHM/fbz2N/p1ahB0WEMdVT+5ebGYTgdeANGCquy8zs/uAHHfPjr0FRgB57r62zPwfAM+Y2SNAAXBDMP8vwfw1RHoSY+PfHRGR5FN8qISfvbycZ+av59y+7XlkbCZNGibObW6WCj/Ws7KyXDWzRSQZ7S46yMS/fszbqwu4ecRX+MHI3tSrVzvjEWa20N2zKmuXOClLRKSO2bBjL+OfWsBn277gl5cNYOzQrmGHVC4lChGRECxc/zk3PZ3DwUMlPD1+KKee1CbskCqkRCEiUsteyt3IXbMW07FFY6Ze/1W6t20adkgxKVGIiNQSd+d3b6zh4X+tZmhGK/547RBaHd8w7LAqpUQhIlILig4e4p6/LebvuflcdnInfnHZABrVTws7rLgoUYiI1LDthfu56ZmFLFz/OXed14vvntk91Dutq0qJQkSkBn2yZQ/jpy1g6+79PPbNk7lwYMewQ6oyJQoRkRryzicFfPe5j2hUP43nbx5OZpeWla+UgJQoRERqwLPz1/OT7GX0aNeUP1+XRecTmoQd0lFTohARqUaHSpyf/2MFU+d9xlm92vK7cYNp1rj2a0hUJyUKEZFqUri/mNumf8wbK7dy/akZ/OjCPtQPqYZEdVKiEBGpBvk79zFhWg6rt+zh/tH9uHZ4RtghVRslChGRY7Q4byc3Tsth74FD/OW6LM7s1S7skKqVEoWIyDF4dekmvv98Lq2Pb8TfvjOMXh2ahR1StVOiEBE5Cu7O4//+lMmvrmJw15ZMuTaLts0ahR1WjVCiEBGpogPFJdz74hJmLszj4kHpPDRmII0bJMfjOI6GEoWISBXs3HuAW55dyPy1O5h0Tg/+6+s9kupxHEdDiUJEJE6fbfuC8U8tYOPn+3jkqkwuGdwp7JBqhRKFiEgc3v90O7c8u5C0esZz3x7GVzNahR1SrVGiEBGpxAs5G7j3xSV0bdWEJ68fStfWyfs4jqOhRCEiUoGSEuehf67i8bmfcvpJbXjs6pNpcVxyP47jaChRiIiUY9+BQ9z+Qi6vLN3MN4d15Wej+tEgBR7HcTSUKEREyti6u4gbn85hycZd/OjCPkw4vVvKX9kUixKFiEiU5fm7mTBtAbv2HeSJa7P4et/2YYcUOiUKEZHAGyu28L3pH9O8cQNm3jKcfuktwg4pIShRiEid5+5MnbeOn/9jOf3SW/Dn67Jo37xx2GEljLhGZsxspJmtMrM1ZnZPjHZjzMzNLCuYvtrMcqNeJWaWGSybG2yzdFm7YP71ZlYQNf/G6thREZHyHDxUwo/+vpT7Zy/n3L4deP7mU5Qkyqi0R2FmacBjwDeAPGCBmWW7+/Iy7ZoBk4APSue5+3PAc8HyAcBL7p4btdrV7p5Tzsc+7+4Tq7ozIiJVsWvfQSb+9SPe+WQbt3ytO3ef14t69eruoHVF4ulRDAXWuPtadz8AzABGl9PufmAyUFTBdsYB048qShGRavaf7Xu5/PH3eP/T7UweM5B7zu+tJFGBeBJFJ2BD1HReMO8wMxsMdHH32TG2cxVfThRPBqeXfmxHXnt2uZktNrNZZtYljhhFROK2cP0OLvnDPAr27OeZCcO4MktfM7HEkyjKS7F+eKFZPeBh4I4KN2A2DNjr7kujZl/t7gOAM4LXtcH8l4EMdx8I/AuYVsE2bzKzHDPLKSgoiGM3RETgpdyNjHviA5o3rs+L3z2V4d1bhx1SwosnUeQB0em2M5AfNd0M6A/MNbN1wClAdumAdmAsZXoT7r4x+HcP8Fcip7hw9+3uvj9o9gQwpLyg3H2Ku2e5e1bbtm3j2A0RqcvcnYdfX81tM3IZ3KUlL373NL7StmnYYSWFeC6PXQD0MLNuwEYiX/rfLF3o7ruANqXTZjYXuLN0kDrocVwBjIhqUx9o6e7bzKwBcBGR3gNm1tHdNwVNRwErjnrvRESAooOHuHvWYrIX5TNmSGcevHQADevXzcdxHI1KE4W7F5vZROA1IA2Y6u7LzOw+IMfdsyvZxAggz93XRs1rBLwWJIk0IkniiWDZJDMbBRQDO4Drq7JDIiLRthXu56anc/joPzu5e2QvvvO17nX6cRxHw9y98lYJLisry3NyyrvKVkTqsk+27OGGpxawrXA/D1+ZyfkDOoYdUkIxs4XunlVZO92ZLSIp6e3VBdz63Ec0bpjG8zcNZ1CXlmGHlLSUKEQk5Twzfz0/zV5Gz/bN+Mt1WaS3PC7skJKaEoWIpIxDJc4D/1jOk/PWcU7vdjw6bjBNG+lr7ljpCIpISijcX8yk6R/z5sqtjD+tG/de2Ic03WldLZQoRCTp5e/cx/inFvDJ1kIeuKQ/15xyYtghpRQlChFJaos27OTGp3MoOnCIJ6//KiN66gbc6qZEISJJa86STdz+Qi5tmzXirzcOo0f7ZmGHlJKUKEQk6bg7f5j7KQ+9toohJ57AlGuH0Lppo7DDSllKFCKSVA4Ul/DfLy5h1sI8Rmem86vLB9K4QVrYYaU0JQoRSRqff3GAm59dyIef7eD7X+/Bbef00OM4aoEShYgkhbUFhYx/agH5u4p4dGwmozM7Vb6SVAslChFJeO99uo3vPPsR9esZ0789jCEntgo7pDpFiUJEEtoLCzbw3y8uoVub45l6/Vfp0qpJ2CHVOUoUIpKQSkqcX722kj/9ey1n9GjDY1efTPPGDcIOq05SohCRhLP3QDH/9Xwury3bwjWndOWnF/ejfpoKDYVFiUJEEsqW3UXcOC2HZfm7+J+L+nLDaRm6silkShQikjCW5e9iwlM57Ck6yJ+vy+Ls3u3DDklQohCRBLE8fzdX/PF9Wh7XgFnfOZU+HZuHHZIElChEJHT7Dhzie9M/ommj+rx462m0b9447JAkihKFiITuvtnLWbvtC56dMExJIgHpMgIRCdWrSzcx/cP/cPOI7px2Upuww5FyKFGISGjyd+7jB39bwqDOLbjj3J5hhyMVUKIQkVAcKnG+/3wuxYdKeHTsYBroPomEpTEKEQnFY2+t4cPPdvCbKwaR0eb4sMORGJTCRaTWLVy/g0ff+ITRmelcdrKeApvolChEpFbt2neQSdNzSW/ZmAcu6a+7rpNAXInCzEaa2SozW2Nm98RoN8bM3MyygumrzSw36lViZpnBsrnBNkuXtQvmNzKz54PP+sDMMo59N0UkEbg79764hM27i3h07GCa6SF/SaHSRGFmacBjwPlAX2CcmfUtp10zYBLwQek8d3/O3TPdPRO4Fljn7rlRq11dutzdtwbzJgCfu/tJwMPAr45y30QkwcxamMfsxZu4/Rs9ObnrCWGHI3GKp0cxFFjj7mvd/QAwAxhdTrv7gclAUQXbGQdMj+PzRgPTgvezgHNMfVORpLe2oJCfZC/jlK+04pavdQ87HKmCeBJFJ2BD1HReMO8wMxsMdHH32TG2cxVfThRPBqedfhyVDA5/nrsXA7uA1mU3ZmY3mVmOmeUUFBTEsRsiEpYDxSXcNiOXhvXr8chVg0mrp99+ySSeRFHef1E/vNCsHpFTRHdUuAGzYcBed18aNftqdx8AnBG8ro3n8w7PcJ/i7lnuntW2bdvK90JEQvPrf65iycZdTL58IB1a6BEdySaeRJEHdIma7gzkR003A/oDc81sHXAKkF06oB0YS5nehLtvDP7dA/yVyCmuIz7PzOoDLYAd8e2OiCSat1cXMOXttVxzSlfO7dch7HDkKMSTKBYAPcysm5k1JPKln1260N13uXsbd89w9wxgPjDK3XPgcI/jCiJjGwTz6ptZm+B9A+AioLS3kQ1cF7wfA7zp7l/qUYhI4ttWuJ/bX1hEj3ZNufeCL10DI0mi0juz3b3YzCYCrwFpwFR3X2Zm9wE57p4dewuMAPLcfW3UvEbAa0GSSAP+BTwRLPsL8IyZrSHSkxhbpT0SkYTg7tw1cxG7iw7y7I1DOa5hWtghyVGK6xEe7j4HmFNm3v9U0PbMMtNziZyOip73BTCkgvWLiPRARCSJPTlvHW+tKuBno/rRu4OKECUz3ZktItVuef5ufvnKSr7epx3fGn5i2OHIMVKiEJFqVVqtrmWTBkweM0iP6EgBenqsiFSr6Gp1rY5vGHY4Ug3UoxCRaqNqdalJiUJEqoWq1aUuJQoROWaqVpfaNEYhIsdM1epSm9K+iBwTVatLfUoUInLUVK2ubtCpJxE5KtHV6mbeMlzV6lKYehQiclRUra7uUKIQkSpTtbq6RYlCRKpE1erqHo1RiEiVlFarm3LtEFWrqyPUoxCRuKlaXd2kRCEicYmuVvejC1Wtri7RqScRqVTZanWNG6haXV2iHoWIVKq0Wt29F/RRtbo6SIlCRGJalr9L1erqOCUKEanQvgOHmDT9Y1Wrq+M0RiEiFVK1OgH1KESkAqpWJ6WUKETkS1StTqIpUYjIEaKr1f1unKrVicYoRKSM0mp1v71yECe2VrU6ibNHYWYjzWyVma0xs3titBtjZm5mWcH01WaWG/UqMbPMMutkm9nSqOmfmtnGqHUuONqdE5Gqia5Wd+lgVauTiEp7FGaWBjwGfAPIAxaYWba7Ly/TrhkwCfigdJ67Pwc8FywfALzk7rlR61wGFJbzsQ+7+6+rvjsicrRUrU4qEk+PYiiwxt3XuvsBYAYwupx29wOTgaIKtjMOmF46YWZNgduBB6oUsYhUu+hqdY+OHaxqdXKEeBJFJ2BD1HReMO8wMxsMdHH32TG2cxVRiYJIYvkNsLecthPNbLGZTTUzlc4SqWGqViexxJMoyut/+uGFZvWAh4E7KtyA2TBgr7svDaYzgZPc/cVymj8OdAcygU1Ekkl527zJzHLMLKegoCCO3RCR8qhanVQmnkSRB3SJmu4M5EdNNwP6A3PNbB1wCpBdOqAdGMuRvYnhwJCg/btATzObC+DuW9z9kLuXAE8QOfX1Je4+xd2z3D2rbdu2ceyGiJSlanUSj3gSxQKgh5l1M7OGRL70s0sXuvsud2/j7hnungHMB0a5ew4c7nFcQWRso3Sdx909PWh/OrDa3c8M2neM+uxLgaWISI0orVY3+fKBqlYnFar0qid3LzazicBrQBow1d2Xmdl9QI67Z8feAiOAPHdfG2dMk4NTUw6sA26Ocz0RqQJVq5N4mbtX3irBZWVleU5OTthhiCSNbYX7GfnIO7Q6vgHZE09XIaI6yswWuntWZe10Z7ZIHaNqdVJVeoiLSB2janVSVUoUInWIqtXJ0VCiEKkj9h4oVrU6OSoaoxCpI+6fvULV6uSoqEchUgeoWp0cCyUKkRSnanVyrJQoRFKYqtVJddAYhUgKU7U6qQ76eSGSokqr1V2Smc5lJ3cOOxxJYkoUIikoulrd/Zf0DzscSXI69SSSYqKr1c28Zbiq1ckxU49CJMXMVLU6qWZKFCIpZG1BIT9VtTqpZkoUIilC1eqkpmiMQiRFlFarm3LtEFWrk2qlHoVIClC1OqlJShQiSW5b4X5uf2ERPds35UcX9g07HElBOvUkksRUrU5qg3oUIkmstFrdjy5UtTqpOUoUIkkqulrdtaeoWp3UHCUKkSSkanVSmzRGIZKEVK1OapN6FCJJRtXqpLYpUYgkEVWrkzAoUYgkCVWrk7DE9ZdmZiPNbJWZrTGze2K0G2NmbmZZwfTVZpYb9Soxs8wy62Sb2dKo6VZm9rqZfRL8q8dfJgh3Z9feg2GHUWeVVqu7/5L+qlYntarSRGFmacBjwPlAX2CcmX3p9k8zawZMAj4onefuz7l7prtnAtcC69w9N2qdy4DCMpu6B3jD3XsAbwTTErJthfu5cVoOg+77JyMfeZs/zF3Dhh17ww6rzlC1OglTPD2KocAad1/r7geAGcDoctrdD0wGiirYzjhgeumEmTUFbgceKNNuNDAteD8NuCSOGKUGzV21lZGPvMM7a7Zxw2kZHNcwjcmvruKMyW9x+ePvMe29dRTs2R92mCmrtFpdp5bHqVqdhCKey2M7ARuipvOAYdENzGww0MXdZ5vZnRVs5yqOTDD3A78Byv4sbe/umwDcfZOZtYsjRqkBRQcP8atXV/LkvHX0bN+UZ28cevju3/9s38vLi/PJzs3nJ9nL+NnLyzjtpDaMGpTOef070FxV1apFdLW6WapWJyGJJ1GUdyePH15oVg94GLi+wg2YDQP2uvvSYDoTOMnd/8vMMqoQb/Q2bwJuAujatevRbEJiWL1lD5Omf8zKzXu4bviJ/PCCPkc8R6hr6ybcetZJ3HrWSazcvJvs3HyyF+Vz16zF3Pv3pZzdqx2jMtM5u3c7PX/oGJRWq7vrvF4MVrU6CUk8iSIP6BI13RnIj5puBvQH5gZ3h3YAss1slLvnBG3GEnXaCRgODDGzdUEM7cxsrrufCWwxs45Bb6IjsLW8oNx9CjAFICsry8trI1Xn7jz9/noenLOCZo3r8+T1X+Ws3rE7db07NKf3yObcdV4vPt6wk+zcfGYv3sSryzbTtFF9zu3XnlGD0jntpDa6UqcKVK1OEoW5x/6ONbP6wGrgHGAjsAD4prsvq6D9XODO0iQR9Dj+A4xw97XltM8AZrt7/2D6IWC7u/8yuMKqlbvfHSvGrKwsz8nJidVE4rCtcD93z1rMmyu3cmavtjw0ZhBtmzU6qm0VHyph/todvJS7kVeXbWZPUTGtjm/IhQM6MioznSFdT6CeKrBV6EBxCZc9Po+8z/fx6m0jVIhIaoSZLXT3rMraVdqjcPdiM5sIvAakAVPdfZmZ3QfkuHt2JZsYAeSVlyQq8EvgBTObQCTBXBHnenIM5q7ayp0zF7O76CA/ubgv15+acUzPD6qfVo/Te7Th9B5teODS/sxdVUB2bj4v5Gzgmfnr6dTyOC4a1JFRg9Lp27G5nlVUxq//uYqlG3erWp0khEp7FMlAPYqjV3TwEL98ZSVPvbeOXu2b8ei4zBp9XHXh/mJeX76Zl3LzeeeTbRwqcbq3PZ7RmZ0YNSidjDa6P+Dt1QV8a+qHXHNKVx64ZEDY4UgKi7dHoURRh63avIfbZkQGrK8/NYN7zu9dqwPPO744wJwlm8helM+Hn+0AYGDnFowalM7Fg9Jp37zu/ZLeVrifkY+8Q6vjG5A98XRdCCA1SolCKlQ6YP3zOSto3rg+D40ZVOmAdU3L37mP2YvzeSk3n2X5uzGDYd1aMTqzE+f370DLJqn/hFR3Z/xTC5j36XayJ56mQkRS45QopFzbCvdz18xFvLWq4JgHrGvKpwWFZOfm8/KifNZu+4IGacaIHm0ZlZnON/q2p0nD1Hw6/tR3P+O+2cu5b3Q/vjU8I+xwpA5QopAveWvV1qC+cjH/fX5vrjvGAeua5u4s3bib7EUbeXnRJjbvLuK4Bml8vW97Rg9KZ0TPtjSsnxqX2y7L38Wlj73HiJ5teOJbWQn930VShxKFHFZ2wPp34wbTq0OzsMOqkpIS58N1O8helM+cJZvYufcgLY5rwPn9OzAqM51h3VqTlqSX2+49UMzFv3+XPUXFvPr9ESpEJLVGiUKAyID1pOkfs2pLOAPWNeFAcQnvrolcbvvP5VvYe+AQ7Zo14qKB6YzOTGdg5xZJ9Yv8h/9vMTMWbODZCcNUiEhqVbXdRyHJyd2Z9t46HnxlJc0b1+fJG77KWb1S47FZDevX4+ze7Tm7d3v2HTjEv1ZsIXtRPs/OX8/UeZ+R0boJFw+KJI2T2iV2z+mVJZuY/uEGbvmaqtVJ4lKPIgUV7NnP3bMiA9Zn9WrL5AQcsK4Ju/Ye5NVlkctt3/90OyUOfTo2Dy637UjnE5qEHeIR8nfu4/xH3yGjdRNmfedUPd5Eap1OPdVRb63cyl2zIgPW917Qh28NPzGpTsNUl627i5i9OJI0cjfsBCDrxBMYlZnOBQM60qZpuInzUIkz7on5LNu4izm3naFCRBIKJYo6JnrAuneHZjw6NvkGrGvK+u1f8PKiyNNtV28pJK2e/d8j0fu1D+XR3b974xN++/pqfnvlIBUiktAoUdQh0QPWN5yWwQ9GJv+AdU1ZuXk3L+VG6mhs3LmPhvXrcU7vdowalM5ZtfRI9IXrd3Dln+Zz8cCOPDJ2cI1/nkhFlCjqgCMHrBvw0BUDU2bAuqa5Ox/9ZyfZuRv5x5JNbCs8QLNG9Tm3X+Ry29O6t6Z+DYwZ7Np3kAsefYe0esY/Jp2uQkQSKl31lOIK9uznrlmLmBsMWD90xaDQz7snEzNjyIknMOTEE/jxRX15f+12XsrN57Wlm/nbR3m0adqQCwZEnm57cjU9Er20Wt2W3UXMVLU6SSLqUSSh0gHrPUXF3HthH649pW4OWNeEooOHIo9EX7SRN1ZsZX9xCZ1aHsfFg9IZNSidPh2bHfWxfiFnA3fPWsxd5/Xi1rNOqubIRapOp55SUNkB69+NG0zP9hqwril7ig7y+vItvJSbz7trIo9E79GuKaMGpTMqM71KVyqtLSjkot+/y8DOLXjuxlOS9i5ySS1KFClm5ebd3DY9VwPWIdleuJ85Szfzcm4+H66LPBJ9UJeWkXs0BnakXYxHoqtanSQqJYoU4e489d46fhEMWP/6ioGcqQHrUG3cuY/ZiyKPRF++KfJI9OFfac2oQemc378jLZocOfbw4JwVTHl7LVOuHcK5/TqEFLXIlylRpIDoAeuze7dj8piBGrBOMGu2FpK9KJ/s3I2s276XBmnG13q2Y1RmOl/v046cdZ+rWp0kLCWKJPfmyi3cNXMxhfs1YJ0M3J0lG3dF6mgszmfL7v00aZhGWj2jY4vGqlYnCUmXxyapooOH+MWcFUx7fz29OzRj+k2naMA6CZgZAzu3ZGDnlvzwgj58+NmOw48PeeSqTCUJSWpKFAlk5ebdTJr+Mau3FDL+tG7cPbKXvmCSUFo9Y3j31gzv3jrsUESqhRJFAnB3npy3jl++GhmwfuqGr2rAWkQShhJFyAr27OfOmYv49+oCzgkGrFtrwFpEEogSRYiiB6zvH92PazRgLSIJSIkiBEUHD/HgnBU8rQFrEUkCShS1bMWm3dw2IzJgPeH0yIB1o/oasBaRxBXXc5TNbKSZrTKzNWZ2T4x2Y8zMzSwrmL7azHKjXiVmlhkse9XMFpnZMjP7o5mlBfN/amYbo9a5oDp2NGwlJc7Udz9j9GPz+HzvQaaNH8qPL+qrJCEiCa/SHkXwBf4Y8A0gD1hgZtnuvrxMu2bAJOCD0nnu/hzwXLB8APCSu+cGi690990WOSk/C7gCmBEse9jdf31Me5ZAtu4p4s6Zi3lbA9YikoTiOfU0FFjj7msBzGwGMBpYXqbd/cBk4M4KtjMOmF464e67o2JoCCT/LeLleGPFFu6eFQxYX9Kfa4Z11YC1iCSVeE49dQI2RE3nBfMOM7PBQBd3nx1jO1cRlSiC9V4DtgJ7iPQqSk00s8VmNtXMTihvY2Z2k5nlmFlOQUFBHLtRu4oOHuJ/XlrKhGk5tGvemNnfO12P4RCRpBRPoijvm+3wr38zqwc8DNxR4QbMhgF73X3pERtxPw/oCDQCzg5mPw50BzKBTcBvytumu09x9yx3z2rbtm0cu1F7VmzazcW/f5en31/PhNO78fdbT6WHrmoSkSQVz6mnPKBL1HRnID9quhnQH5gb/FruAGSb2Sh3L31S31jK9CZKuXuRmWUTOZ31urtvKV1mZk8AsXopCaWkxHnyvXX86pWVtGjSgKfHD2VEz8RKYiJbnkaDAAAGsklEQVQiVRVPolgA9DCzbsBGIl/63yxd6O67gDal02Y2F7izNEkEPY4rgBFRbZoCzdx9k5nVBy4A3gmWdXT3TUHTS4EjeiGJKnrA+ut92vGryzVgLSKpodJE4e7FZjYReA1IA6a6+zIzuw/IcffsSjYxAsgrHQwPHE+k19Eo2OabwB+DZZODS2gdWAfcXJUdCsMbK7Zw16zFfKEBaxFJQapHcQyKDh7i5/9YwTPz19OnY3N+Py6Tk9ppLEJEkoPqUdSw5fmRO6w/2VrIjad34y7dYS0iKUqJoopKSpyp8z5j8qurNGAtInWCEkUVbN1dxJ2zSges2zN5zEBaHd8w7LBERGqUEkWcSges9x4o5oFL+nO1BqxFpI5QoqjEvgORR4I/M389fTs253casBaROkaJIobl+buZNONj1mwt5NtndOPO8zRgLSJ1jxJFOaIHrFs2acAzE4ZyRg8NWItI3aREUcbW3UXcMXMR73yyTQPWIiIoURzhX8u3cPffIgPWP7+0P98cqgFrERElCiID1j+fs5xn5/9HA9YiImXU+UQRPWB904ivcMe5PTVgLSISpU4nipk5G7j3xaW0bNKAZycM4/QebSpfSUSkjqnTiaJbm+M5u3c7HrxsgAasRUQqUKcTRVZGK7IyWoUdhohIQounFKqIiNRhShQiIhKTEoWIiMSkRCEiIjEpUYiISExKFCIiEpMShYiIxKREISIiMZm7hx3DMTOzAmD9Ua7eBthWjeFUF8VVNYqr6hI1NsVVNccS14nuXmmxnZRIFMfCzHLcPSvsOMpSXFWjuKouUWNTXFVTG3Hp1JOIiMSkRCEiIjEpUcCUsAOogOKqGsVVdYkam+KqmhqPq86PUYiISGzqUYiISEwpnSjMbKSZrTKzNWZ2TznLrzezAjPLDV43Ri27zsw+CV7XJVBch6LmZ9dmXEGbK81suZktM7O/Rs0P7XhVEldox8vMHo767NVmtjNqWZh/X7HiCvN4dTWzt8zsYzNbbGYXRC37YbDeKjM7LxHiMrMMM9sXdbz+WMtxnWhmbwQxzTWzzlHLqvfvy91T8gWkAZ8CXwEaAouAvmXaXA/8bznrtgLWBv+eELw/Iey4gmWFIR6vHsDHpccCaJcgx6vcuMI+XmXafw+YmgjHq6K4wj5eRM61fyd43xdYF/V+EdAI6BZsJy0B4soAloZ4vGYC1wXvzwaeqam/r1TuUQwF1rj7Wnc/AMwARse57nnA6+6+w90/B14HRiZAXDUpnri+DTwWHBPcfWswP+zjVVFcNamq/x3HAdOD92Efr4riqknxxOVA8+B9CyA/eD8amOHu+939M2BNsL2w46pJ8cTVF3gjeP9W1PJq//tK5UTRCdgQNZ0XzCvr8qDrNsvMulRx3dqOC6CxmeWY2Xwzu6SaYoo3rp5ATzObF3z+yCqsG0ZcEO7xAiKnCIj8En6zquvWclwQ7vH6KXCNmeUBc4j0duJdN4y4ALoFp6T+bWZnVFNM8ca1CLg8eH8p0MzMWse5bpWkcqKwcuaVvcTrZSDD3QcC/wKmVWHdMOIC6OqRuzC/CTxiZt1rMa76RE7znEnkl+ifzaxlnOuGEReEe7xKjQVmufuho1i3qo4lLgj3eI0DnnL3zsAFwDNmVi/OdcOIaxOR4zUYuB34q5k1p3rEE9edwNfM7GPga8BGoDjOdasklRNFHhD9S7wzZbqM7r7d3fcHk08AQ+JdN6S4cPf84N+1wFxgcG3FFbR5yd0PBqcAVhH5gg71eMWIK+zjVWosR57eCft4VRRX2MdrAvBC8PnvA42JPMco7ONVblzBqbDtwfyFRMYUetZWXO6e7+6XBYnq3mDerjj3qWpqYiAmEV5EfmWuJdK1Lh0M6lemTceo95cC8/3/BoM+IzIQdELwvlUCxHUC0Ch43wb4hBgDlTUQ10hgWtTnbwBaJ8DxqiiuUI9X0K4XsI7gnqVE+PuKEVfYf1+vANcH7/sQ+XIzoB9HDmavpfoGs48lrralcRAZdN5Yy3/3bYB6wfufA/fV1N/XMe9QIr+IdBNXE8n09wbz7gNGBe9/ASwL/iO8BfSOWnc8kUGzNcANiRAXcCqwJJi/BJhQy3EZ8FtgefD5YxPkeJUbV9jHK5j+KfDLctYN7XhVFFfYx4vI4Oy84PNzgXOj1r03WG8VcH4ixEVkfKD0/9OPgItrOa4xRJL5auDPBEm+Jv6+dGe2iIjElMpjFCIiUg2UKEREJCYlChERiUmJQkREYlKiEBGRmJQoREQkJiUKERGJSYlCRERi+v9wemxS1MpAEQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "train_means = grid_search.cv_results_['mean_train_score']\n",
    "train_stds = grid_search.cv_results_['std_train_score']\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### colsample_bytree微调"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:25:27.880936Z",
     "start_time": "2020-06-25T08:20:28.683625Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 3 candidates, totalling 9 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   4 out of   9 | elapsed:  2.4min remaining:  3.0min\n",
      "[Parallel(n_jobs=4)]: Done   6 out of   9 | elapsed:  4.4min remaining:  2.2min\n",
      "[Parallel(n_jobs=4)]: Done   9 out of   9 | elapsed:  5.0min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=1.0, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=40,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=380,\n",
       "        n_jobs=4, num_class=9, num_leaves=60, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.8, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.2, 0.3, 0.4]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':60,\n",
    "          'min_child_samples':40,\n",
    "          'max_bin': 127, \n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i / 10.0 for i in range(2, 5)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg,\n",
    "                           n_jobs=4,  \n",
    "                           param_grid=tuned_parameters, \n",
    "                           cv = kfold, \n",
    "                           scoring=\"neg_log_loss\", \n",
    "                           verbose=5, \n",
    "                           refit = False,\n",
    "                          )\n",
    "grid_search.fit(x_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:25:57.567999Z",
     "start_time": "2020-06-25T08:25:57.564010Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.473547974572\n",
      "{'colsample_bytree': 0.4}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:26:34.183677Z",
     "start_time": "2020-06-25T08:26:33.967201Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAD8CAYAAABZ/vJZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VPW5x/HPk5UtrAl7IGET2QwysgUQt4q2sggqiIgrIlB7r7a3ent7b69201Zte9kEFa0iqCiS0ip1ISggS5BFdpIQIKwBZA0kJHnuH3OGHmKWyTpZnvfrNa/M+Z3f+c0zwyRfzpw55yeqijHGGFNSQYEuwBhjTPVkAWKMMaZULECMMcaUigWIMcaYUrEAMcYYUyoWIMYYY0rFAsQYY0ypWIAYY4wpFQsQY4wxpRIS6AIqUmRkpMbExAS6DGOMqVY2bNhwXFWjiutXowMkJiaGpKSkQJdhjDHViojs86effYRljDGmVCxAjDHGlIoFiDHGmFKxADHGGFMqfgWIiAwTkV0ikiwiTxfRb4yIqIh4nOXxIrLJdcsTkThnXaIzpm9dc6f9ARHJcLU/4hp/oojscW4Ty/bUjTHGlEWx38ISkWBgBnALkA6sF5EEVd2er18E8ASw1temqvOB+c76nsASVd3k2my8qhb0Nal3VXVavvGbAv8DeAAFNjh1fFf80zTGGFPe/NkD6Qskq2qqqmYDC4ERBfR7DngBuFjIOOOABaWq0utW4FNVPemExqfAsDKMZ4wxpgz8CZA2wAHXcrrTdpmI9AaiVXVpEePcw/cDZJ7zMdUvRURc7aNFZIuILBKRaH/rKC95ecpv/7GDAyczK2J4Y4ypEfwJECmg7fJE6iISBLwMPFXoACL9gExV3epqHq+qPYHBzm2C0/43IEZVewGfAW/6U4frsSaJSJKIJGVkZBT+rIqQduI8C9ftZ8SMVaxNPVGqMYwxpqbzJ0DSgWjXclvgkGs5AugBJIpIGtAfSPAdSHeMJd/eh6oedH6eBd7B+1EZqnpCVbOcbnOBPn7W4Rt3jqp6VNUTFVXsmfgF6hDVgI+mxtO4XijjX13LgnX7SzWOMcbUZP4EyHqgs4jEikgY3jBI8K1U1dOqGqmqMaoaA6wBhvsOjjt7KHfhPXaC0xYiIpHO/VDgR8BWZ7mV67GHAzuc+8uAH4hIExFpAvzAaasQHaIasHhKPPGdInnmw2/5VcI2cnLzKurhjDGm2in2W1iqmiMi0/D+sQ4GXlfVbSLyLJCkqglFj8AQIF1VU11t4cAyJzyC8X5UNddZ94SIDAdygJPAA04dJ0XkObyBBvCsqp7050mWVqO6obz+wHX87h87eHXlXpKPnWPGvdfSqF5oRT6sMcZUC6L6vcMINYbH49Hyupjie0kH+K/FW2nduA6vTryOTs0blMu4xhhT1YjIBlX1FNfPzkT3092eaBZM6se5rBxGzVjF8l3HAl2SMcYElAVICfRp35Ql0wbRtmk9Hn5jPa9+lUpN3oMzxpiiWICUUJvGdfng8QHc2r0lv/77Dn62aAtZObmBLssYYyqdBUgp1AsLYca91/KTmzqzaEM6985dS8bZrOI3NMaYGsQCpJSCgoR/v6ULM+69lm2HTjNi+kq2Hjwd6LKMMabSWICU0Q97tWLR5IEocNfsr/n428OBLskYYyqFBUg56NGmEUumxXN1qwgen/8Nf/psN3l5dnDdGFOzWYCUk+YRdVgwqT+jr23Lnz7bw7QF35CZnRPosowxpsIUeya68V94SDB/vKsXXVtG8LuPd7DvRCZz7vfQpnHdQJdmjDHlzvZAypmI8OiQDrz2wHXsP5HJiOkr2bCvQq+4YowxAWEBUkFuuKo5i6cOpEF4COPmrOX9pAPFb2SMMdWIBUgF6tQ8go+mxnNdbBN+tmgLv/n7dnLt4LoxpoawAKlgjeuF8caDfZk4oD1zv9rLw2+u58zFS4EuyxhjyswCpBKEBgfxvyN68NtRPVm55zijZqxi7/HzgS7LGGPKxAKkEt3brx1vP9KPk+ezGTljFSv3HA90ScYYU2oWIJWsf4dmJEwbRMuGdZg4bx1vrk6zK/oaY6olvwJERIaJyC4RSRaRp4voN0ZE1DcfuoiMF5FNrlueiMQ56xKdMX3rmjvtT4rIdhHZIiKfi0h71/i5rv7FzYRYZUU3rccHUwZyw1XN+Z+Ebfzn4q1k59h0ucaY6qXYABGRYGAGcBvQDRgnIt0K6BcBPAGs9bWp6nxVjVPVOGACkKaqm1ybjfetV1XfDE0bAY+q9gIWAS+4+l9w9R9esqdatTQID2HOhD5MGdqRBev2c99razl5PjvQZRljjN/82QPpCySraqqqZgMLgREF9HsO7x/7i4WMMw5YUNyDqepyVc10FtcAbf2osVoKChL+Y1hX/jw2jk0HTjF8+kp2HjkT6LKMMcYv/gRIG8B9Fly603aZiPQGolV1aRHj3MP3A2Se83HUL0VECtjmYeBj13IdEUkSkTUiMtKP2quFEXFteP+xAWTn5DF65mr+ue1IoEsyxphi+RMgBf1hv3zUV0SCgJeBpwodQKQfkKmqW13N41W1JzDYuU3It819gAf4g6u5nTPR+73An0SkYwGPNckJmaSMjIxin1xVcU10Y/7240F0at6Ax97ewIzlyXZw3RhTpfkTIOlAtGu5LXDItRwB9AASRSQN6A8k+A6kO8aSb+9DVQ86P88C7+D9qAwAEbkZ+AUwXFWzXNsccn6mAolA7/zFquocVfWoqicqKsqPp1d1tGhYh3cfG8AdvVrzh2W7+MnCTVy8ZNPlGmOqJn8CZD3QWURiRSQMbxhc/gaUqp5W1UhVjVHVGLzHLYarahJc3kO5C++xE5y2EBGJdO6HAj8CtjrLvYFXnDGOubZpIiLhzv1IIB7YXupnXkXVCQ3mz2Pj+NmtV/G3LYe4+5WvOXK6sMNKxhgTOMUGiKrmANOAZcAO4D1V3SYiz4qIP9+EGgKkO3sNPuHAMhHZAmwCDgJznXV/ABoA7+f7uu7VQJKIbAaWA79X1RoXIOC9ou/UGzoxZ4KHlGPnGD59JZsOnAp0WcYYcwWpyZ+zezweTUpKCnQZZbLzyBkeeTOJY2ez+MOYXoyIa1P8RsYYUwYissE53lwkOxO9iuvasiFLpsYTF92YnyzcxPOf7LTpco0xVYIFSDXQrEE4bz/cj3F92zErMYVJb23gXJZNl2uMCSwLkGoiLCSI347qwf8O787yXccYPXM1B05mFr+hMcZUEAuQakREmDgwhr8+1JcjZy4yfPpK1qSeCHRZxphaygKkGorvFMlHU+NpWj+M+15dy/y1+wJdkjGmFrIAqaZiI+uzeGo8gzpH8ovFW/nvJVu5lGtX9DXGVB4LkGqsYZ1QXpt4HZOGdOCvX+/jgXnrOJVpV/Q1xlQOC5BqLjhI+M/br+aPd13D+r3fMXLGKpKPnQ10WcaYWsACpIYY06ctCyb141xWLqNmrGb5zmPFb2SMMWVgAVKD9GnflIRp8bRrVo+H3lzPnC9T7Iq+xpgKYwFSw7RuXJf3Jw/gth4t+e0/dvLT97fYFX2NMRXCAqQGqhcWwvRx1/JvN3fmg2/SGTd3DcfO2hV9jTHlywKkhgoKEv7t5i7MGn8tOw+fZcT0VWw9eDrQZRljahALkBrutp6tWPT4AAQYM3s1f99yONAlGWNqCAuQWqB760YsmTaI7q0bMfWdb3jp0912RV9jTJlZgNQSURHhvPNoP+7q05a/fL6Hqe98Q2a2XdHXGFN6FiC1SHhIMC+M6cV//fBqlm07wuhZX5P+nV3R1xhTOn4FiIgME5FdIpIsIk8X0W+MiKiIeJzl8c60tL5bnojEOesSnTF965o77eEi8q7zWGtFJMY1/jNO+y4RubUsT7y2EhEeGdyB1x+4jvSTmYycsYqktJOBLssYUw0VGyAiEgzMAG4DugHjRKRbAf0igCeAtb42VZ2vqnGqGgdMANJUdZNrs/G+9arqO3X6YeA7Ve0EvAw874zfDRgLdAeGATOd2kwpDL2qOYunxtMgPIRxc9fwXtKBQJdkjKlm/NkD6Qskq2qqqmYDC4ERBfR7DngBKOyEg3HAAj8ebwTwpnN/EXCTiIjTvlBVs1R1L5Ds1GZKqVPzBiyZOoh+sc34j0VbeG7pdnLsir7GGD/5EyBtAPd/T9OdtstEpDcQrapLixjnHr4fIPOcj69+6YTEFY+nqjnAaaCZP3U4tUwSkSQRScrIyCj2ydV2jeqF8saD1/HAwBheW7mXh95M4vSFS4EuyxhTDfgTIFJA2+XvgIpIEN6Pmp4qdACRfkCmqm51NY9X1Z7AYOc2oZjHK7KOyw2qc1TVo6qeqKiowkoyLiHBQfxqeHd+d2dPVicfZ9TMVaRmnAt0WcaYKs6fAEkHol3LbYFDruUIoAeQKCJpQH8gwXcg3TGWfHsfqnrQ+XkWeId/fRx1+fFEJARoBJz0ow5TRuP6tmP+I/04lXmJkTNW8dUe24MzxhTOnwBZD3QWkVgRCcMbBgm+lap6WlUjVTVGVWOANcBwVU2Cy3sod+E9doLTFiIikc79UOBHgG/vJAGY6NwfA3yh3kvKJgBjnW9pxQKdgXWlfN6mEP06NGPJ1HhaN67LA/PWM2/VXruirzGmQMUGiHMcYhqwDNgBvKeq20TkWREZ7sdjDAHSVTXV1RYOLBORLcAm4CAw11n3GtBMRJKBJ4GnnTq2Ae8B24FPgKmqapeZrQDRTeux6PGB3Ni1Of/7t+088+G3ZOfYwXVjzJWkJv/v0uPxaFJSUqDLqLby8pSXPt3N9OXJ9I1pyqz7rqVZg/BAl2WMqWAiskFVPcX1szPRTaGCgoSf3noVfx4bx+b0Uwyfvoodh88EuixjTBVhAWKKNSKuDe89NoCcvDxGz1rNsm1HAl2SMaYKsAAxfrkmujEJ0wbRuUUEj721gelf7LGD68bUchYgxm8tGtbh3Un9GRnXmj/+czdPLNzEhWz7HoMxtVVIoAsw1Uud0GBevieOLi0j+MOyXaQdP8/c+z20bFQn0KUZYyqZ7YGYEhMRpgztxNwJHlIzznHH9JVs3P9doMsyxlQyCxBTajd3a8GHU+KpExrEPXPWsHhjeqBLMsZUIgsQUyZXtYxgydRBXNuuMf/+7mZ+//FOcm26XGNqBQsQU2ZN64fx1sP9GN+vHbNXpDDpr0mcvWhX9DWmprMAMeUiNDiI34zqyXMjupO4O4PRs1az/4RNl2tMTWYBYsrVhAEx/PWhvhw9k8XwGStZnXI80CUZYyqIBYgpd/GdIlkyNZ7IBuHc/9o63lqzL9AlGWMqgAWIqRAxkfX5cMpABneO5JcfbeWXH23lkk2Xa0yNYgFiKkzDOqG8OvE6HhvSgbfW7OP+19bx3fnsQJdljCknFiCmQgUHCc/cfjUv3nUNG/Z9x8iZq9hz9GygyzLGlAMLEFMpRvdpy4JJ/Tmflcuomav5YufRQJdkjCkjvwJERIaJyC4RSRaRp4voN0ZE1DcfuoiMF5FNrlueiMTl2yZBRLa6lt919U8TkU1Oe4yIXHCtm126p2wCpU/7JiRMiycmsh4Pv5nEKytS7Iq+xlRjxV5MUUSCgRnALUA6sF5EElR1e75+EcATwFpfm6rOB+Y763sCS1R1k2ubO4Fz7nFU9R7X+heB067VKap6RQCZ6qV147q8/9hAfrpoM7/7eCe7jpzlt3f2pE5ocKBLM8aUkD97IH2BZFVNVdVsYCEwooB+zwEvABcLGWccsMC3ICIN8M55/uuCOouIAHe7tzE1Q92wYKaP682Tt3Thw40HGTtnDcfOFPa2McZUVf4ESBvggGs53Wm7TER6A9GqurSIce7hyjB4DngRKOx05cHAUVXd42qLFZGNIrJCRAb7UbupokSEJ27qzKzx17LryFmGT1/Ft+mni9/QGFNl+BMgUkDb5Q+uRSQIeBl4qtABRPoBmaq61VmOAzqp6uIiHveKPRbgMNBOVXvj3XN5R0QaFvBYk0QkSUSSMjIyihjeVAW39WzFoscHEBwk3PXKav62+VCgSzLG+MmfAEkHol3LbQH3b3kE0ANIFJE0oD+Q4DuQ7hjLlWEwAOjj9F8JdBGRRN9KEQkB7gTe9bWpapaqnnDubwBSgC75i1XVOarqUVVPVFSUH0/PBFr31o1YMi2eHq0b8eMFG3npn7vIsyv6GlPl+RMg64HOIhIrImF4wyDBt1JVT6tqpKrGqGoMsAYYrqpJcHkP5S68x05828xS1dZO/0HAblUd6nrMm4Gdqnp5ggkRiXIO6CMiHYDOQGopnrOpgiIbhDP/0X7c7WnLX75I5vH5GziflRPosowxRSg2QFQ1B5gGLAN2AO+p6jYReVZEhvvxGEOAdFUtyR/7/HssvnG2iMhmYBEwWVVPlmBMU8WFhwTz/Ohe/PJH3fh0+1FGz1pN+nd2RV9jqiqpyd/D93g8mpSUFOgyTCms2J3BtHe+ISw4iNkT+nBdTNNAl2RMrSEiG1TVU1w/OxPdVEnXd4nio6nxNKwbyr1z1/Du+v2BLskYk48FiKmyOkY14KMp8fTv0Iyff/At//u3beTYFX2NqTIsQEyV1qheKPMeuI4H42OYtyqNB99Yz+kLNl2uMVWBBYip8kKCg/ifO7rz/OierEk9wagZq0jJOFf8hsaYCmUBYqqNe65rx/xH+nPqwiVGzljFl7vtRFFjAskCxFQrfWObsmRqPG0a1+WBeet4beVeu6KvMQFiAWKqneim9fjg8YHcfHULnlu6nZ9/sIWsnNxAl2VMrWMBYqql+uEhzL6vDz++sRPvJaVz36trOX4uK9BlGVOrWICYaisoSHjqB1fxf+N6syX9NCOmr2L7oTOBLsuYWsMCxFR7d1zTmkWTB5Kbp4yetZpPth4JdEnG1AoWIKZG6Nm2EQnT4rmqZQST397AXz7fYwfXjalgFiCmxmjesA4LJ/Xnzt5teOnT3UxbsJEL2XZw3ZiKUuyc6MZUJ3VCg3nx7mvo0jKC5z/Zyb4T55l7v4dWjeoGujRjahzbAzE1jogw+fqOvHq/h7TjmQyfvopv9n8X6LKMqXEsQEyNddPVLfhwykDqhgYzds4aPvwmvfiNjDF+swAxNVqXFhEsmRrPte0a8+R7m/ndP3aQa9PlGlMuLEBMjdekfhhvPdyP+/q345UvU3n0r0mcvWhX9DWmrPwKEBEZJiK7RCRZRJ4uot8YEVER8TjL40Vkk+uWJyJx+bZJEJGtruVfichB1za3u9Y949SwS0RuLfnTNbVVaHAQvx7Zk+dG9mDF7gxGzVxN2vHzgS7LmGqt2AARkWBgBnAb0A0YJyLdCugXATwBrPW1qep8VY1T1ThgApCmqptc29wJFHRd7pd926nqP5y+3fDOld4dGAbMdGozxm8T+rfnrYf7cvxcFiNnrmJ18vFAl2RMteXPHkhfIFlVU1U1G1gIjCig33PAC8DFQsYZByzwLYhIA+BJ4Nd+1joCWKiqWaq6F0h2ajOmRAZ2jGTJ1HiiGoQz4fV1vPV1WqBLMqZa8idA2gAHXMvpTttlItIbiFbVpUWMcw+uAMEbOC8CmQX0nSYiW0TkdRFp4m8dTi2TRCRJRJIyMmy+CFOw9s3q8+GUgQztEsUvl2zjF4u/5ZJNl2tMifgTIFJA2+WvsYhIEPAy8FShA4j0AzJVdauzHAd0UtXFBXSfBXQE4oDDeEOm2DouN6jOUVWPqnqioqIKK8kYIuqEMud+D49d34H5a/cz4bW1fHc+O9BlGVNt+BMg6UC0a7ktcMi1HAH0ABJFJA3oDyT4DqQ7xnLl3scAoI/TfyXQRUQSAVT1qKrmqmoeMJd/fUxVXB3GlFhwkPDMbVfz0t3X8M3+UwyfsZLdR88GuixjqgV/AmQ90FlEYkUkDG8YJPhWquppVY1U1RhVjQHWAMNVNQku76HchffYiW+bWara2uk/CNitqkOd/q1cjz0K8H1DKwEYKyLhIhILdAbWleI5G/M9d17blncn9efipTxGzVjFZ9uPBrokY6q8YgNEVXOAacAyYAfwnqpuE5FnRWS4H48xBEhX1VQ/a3pBRL4VkS3ADcC/O3VsA94DtgOfAFNV1a6UZ8pN73ZNSJgWT2xUfR59K4lZiSl2RV9jiiA1+RfE4/FoUlJSoMsw1cyF7Fx+tmgzS7ccZmRca34/uhd1Qu0b46b2EJENquoprp9djdeYfOqGBfN/43pzVYsIXvx0N3tPZDJ3Qh+aN6wT6NKMqVLsUibGFEBE+PFNnZl9Xx/2HD3LHdNXsiX9VKDLMqZKsQAxpgjDerTkg8cHEhIUxF2zvyZhs33xzxgfCxBjinF1q4YsmRZPr7aNeGLBRv64bBd5dkVfYyxAjPFHZINw5j/Sn3s80UxfnszktzdwPisn0GUZE1AWIMb4KSwkiN+P7sn/3NGNz3YcZfSs1Rw4WdCVeIypHSxAjCkBEeHB+FjeeLAvB09dYMSMVaxNPRHosowJCAsQY0phSJcolkyNp3HdUO57bS0L1+0PdEnGVDoLEGNKqUNUAxZPjWdAx0ie/vBbfpWwjRy7oq+pRSxAjCmDRnVDeX2ih4fiY3ljdRoPvrGe05k2Xa6pHSxAjCmjkOAg/vuObrwwuhdrUk8wcuYqko8VNNGmMTWLBYgx5eTu66J559H+nLlwiVEzV5G461igSzKmQlmAGFOOrotpypJp8bRtUo+H3ljPq1+l2hV9TY1lAWJMOWvbpB6LJg/glm4t+PXfd/CzRVvIyrGZB0zNYwFiTAWoHx7CrPF9eOKmzizakM69c9eScTYr0GUZU64sQIypIEFBwpO3dGH6vb3Zdug0I6avZNuh04Euy5hy41eAiMgwEdklIski8nQR/caIiPrmQxeR8SKyyXXLE5G4fNskiMhW1/IfRGSniGwRkcUi0thpjxGRC66xZpfuKRtTuX7UqzWLJg9EgTGzvubjbw8HuiRjykWxASIiwcAM4DagGzBORLoV0C8CeAJY62tT1fmqGqeqccAEIE1VN7m2uRPI/33HT4EeqtoL2A0841qX4htPVSf7+ySNCbQebRqxZFo8XVtF8Pj8b/jzZ3vs4Lqp9vzZA+kLJKtqqqpmAwuBEQX0ew54AbhYyDjjgAW+BRFpADwJ/NrdSVX/6czDDrAGaOtHjcZUec0j6rDg0f7ceW0bXv5sN3fN/prEXccsSEy15U+AtAEOuJbTnbbLRKQ3EK2qS4sY5x5cAYI3cF4Eirqc6UPAx67lWBHZKCIrRGSwH7UbU6XUCQ3mxbuu4bejenLw1AUemLeeH/3fSpZuOUSuzTFiqhl/AkQKaLv8TheRIOBl4KlCBxDpB2Sq6lZnOQ7opKqLi9jmF0AOMN9pOgy0U9XeePdc3hGRhgVsN0lEkkQkKSMjo9gnZ0xlExHu7deOFT+7gRfG9OJCdi7T3tnIzS+t4N31+8nOsetpmerBnwBJB6Jdy20B97yeEUAPIFFE0oD+QILvQLpjLFfufQwA+jj9VwJdRCTRt1JEJgI/Asars3+vqlmqesK5vwFIAbrkL1ZV56iqR1U9UVFRfjw9YwIjLCSIuz3RfPrk9cwcfy31woL5+QffMuSF5by2ci+Z2TZhlanapLjPX0UkBO/B7JuAg8B64F5V3VZI/0Tgp6qa5CwHAfuBIaqaWkD/GGCpqvZwlocBLwHXq2qGq18UcFJVc0WkA/AV0FNVTxZWu8fj0aSkpCKfnzFVhary5Z7jzFyezNq9J2lSL5QH42OZOCCGRvVCA12eqUVEZIOqeorrF1JcB1XNEZFpwDIgGHhdVbeJyLNAkqomFDPEECC9oPAoxHQgHPhURADWON+4GgI8KyI5QC4wuajwMKa6ERGu7xLF9V2i2LDvJDOXp/DSp7t5ZUUK4/u355FBsTRvWCfQZRpzWbF7INWZ7YGY6m7H4TPMSkxh6ZZDhAQFMcbTlslDOtKuWb1Al2ZqMH/3QCxAjKkG9p04zytfprIoKZ2cvDzuuKY1jw/tSNeW3/seiTFlZgGCBYipeY6euchrK/fy9pp9ZGbnclPX5ky5oRN92jcJdGmmBrEAwQLE1FynMrN5c/U+5q3ey6nMS/SLbcqUGzoxpHMkzrFDY0rNAgQLEFPznc/KYcG6/bz61V6OnLlIjzYNmTK0E7d2b0lwkAWJKR0LECxATO2RlZPLRxsPMisxhbQTmXSIqs/k6zsyMq4NYSF20W1TMhYgWICY2ic3T/l462FmLk9h++EztG5Uh0eHdGDsde2oGxYc6PJMNWEBggWIqb1UlcTdGcxansK6tJM0rR/GgwNjuN9OSjR+sADBAsQYgPVpJ5m5PJnluzJoEB7C+P7teHhQLM0j7KREUzALECxAjHHbdug0sxJT+Me3hwkJDuJuT1seG9KR6KZ2UqK5kgUIFiDGFCTt+Hle+TKFRRvSyVMY7pyU2KVFRKBLM1WEBQgWIMYU5cjpi7z6VSrvrNtPZnYuN1/dgik3dOTadnZSYm1nAYIFiDH++O58Nm+sTuON1WmcvnCJAR2aMeWGjgzqZCcl1lYWIFiAGFMS57JyWLB2P3O/SuXY2Sx6tW3ElKEd+UG3lgTZSYm1igUIFiDGlEZWTi4ffnOQ2StS2Hcik45R9Xl8aCdGxLUmNNhOSqwNLECwADGmLHJy8/jH1iPMXJ7MziNnadO4LpOGdOBuT7SdlFjDWYBgAWJMeVBVlu86xszlKSTt+45m9cN4aFAs9/VvT6O6dlJiTeRvgPi1Pyoiw0Rkl4gki8jTRfQbIyLqmw9dRMaLyCbXLU9E4vJtkyAiW13LTUXkUxHZ4/xs4rSLiPzFqWGLiFzrT+3GmLIREW7s2oJFjw/kvccG0KNNI/6wbBeDfv8Fz3+yk4yzWYEu0QRIsQEiIsHADOA2oBswTkS6FdAvAngCWOtrU9X5qhqnqnHABCBNVTe5trkTOJdvqKeBz1W1M/C5s4zz+J2d2yRglr9P0hhTPvrGNuXNh/qy9MeDGNIlitkrUhj0/Bf895KtHDiZGejyTCXzZw+kL5Csqqmqmg0sBEYU0O854AXgYiHjjAMW+BZEpAHwJPDrfP1GAG86998ERrra/6pea4BuWHafAAASYklEQVTGItLKj/qNMeWsR5tGzBh/LZ8/eT0j49qwYN1+hv4xkSff3cSeo2cDXZ6pJP4ESBvggGs53Wm7TER6A9GqurSIce7BFSB4A+dFIP9/W1qo6mEA52dzf+swxlSuDlENeH5ML778jxuYOCCGj7ce4ZaXv2TSX5PYdOBUoMszFcyfACnoC+CXj7yLSBDwMvBUoQOI9AMyVXWrsxwHdFLVxSWotcg6XI81SUSSRCQpIyOjBMMbY0qrVaO6/Pcd3Vj19I08cWMn1qSeYOSMVYx/dQ2rk49Tk7+sU5v5EyDpQLRruS1wyLUcAfQAEkUkDegPJPgOpDvGcuXexwCgj9N/JdBFRBKddUd9H005P4/5WQcAqjpHVT2q6omKivLj6RljykvT+mE8+YOrWP3MTfzn7V3ZffQc9766lpEzV7Ns2xHy8ixIahJ/AmQ90FlEYkUkDG8YJPhWquppVY1U1RhVjQHWAMNVNQku76HchffYiW+bWara2uk/CNitqkOd1QnAROf+RGCJq/1+59tY/YHTvo+6jDFVS4PwECYN6chX/3EDvxnVg5Pns3jsrQ3c+qcv+fCbdC7l5gW6RFMOig0QVc0BpgHLgB3Ae6q6TUSeFZHhfjzGECBdVVP9rOn3wC0isge4xVkG+AeQCiQDc4Epfo5njAmQOqHBjO/XnuVPDeXPY+MIEuHJ9zZzwx8TeevrNC5eyg10iaYM7ERCY0ylyctTvth5jJmJyXyz/xSRDcJ5eFAs9/VvR0QdOymxqrAz0bEAMaaqUlXW7j3JjOXJfLXnOBF1Qrh/QHsejI8lskF4oMur9SxAsAAxpjr4Nv00s1Yk8/HWI4SHBDH2unY8OqQDbRrXDXRptZYFCBYgxlQnycfO8cqKFBZvPAjAyN5tmHx9Rzo1bxDgymofCxAsQIypjg6eusDcL1NZuH4/WTl5DOvekilDO9GzbaNAl1ZrWIBgAWJMdXbiXBbzVqXx5tdpnL2Yw+DOkUwZ2on+HZraTIkVzAIECxBjaoKzFy/x9pr9vLZyL8fPZdG7XWOmDO3ETV2b20yJFcQCBAsQY2qSi5dyeT/pAK98mUr6dxe4qkUEjw/tyI96tSLEZkosVxYgWIAYUxNdys1j6ZZDzFyewp5j54huWpfHhnRkTJ+21Am1mRLLgwUIFiDG1GR5ecpnO44yIzGFzQdOERURziODYhnfvz0NwkMCXV61ZgGCBYgxtYGq8nXKCWYmprAy+TgN64QwcWAMD8bH0rR+WKDLq5YsQLAAMaa22XzgFDMTk1m27Sh1Q4MZ2zeaRwd3oLWdlFgiFiBYgBhTW+05epZZK1JYsukQQQKjnJMSO0TZSYn+sADBAsSY2i79u0znpMQDZOfmcVsP70mJPdrYSYlFsQDBAsQY45VxNot5q/by1tf7OJuVw5AuUUwd2pG+sXZSYkEsQLAAMcZc6czFS7z19T5eX7mXE+ez6dO+CVOGduTGrs0tSFwsQLAAMcYU7OKlXN5LOsArK1I5eOoCXVt6T0r8YU87KREsQAALEGNM0S7l5pGw6RCzVqSQfOwc7ZrWY/L1HRndpw3hIbX3pER/A8SvqBWRYSKyS0SSReTpIvqNEREVEY+zPF5ENrlueSIS56z7REQ2i8g2EZktIsFO+7uu/mkisslpjxGRC651s/2p3RhjChMaHMToPm35578NYfZ9fWhSL5T/XPwtg59fzpwvUziXlRPoEqu0YvdAnD/su/HOT54OrAfGqer2fP0igL8DYcA0VU3Kt74nsERVOzjLDVX1jHg/eFwEvK+qC/Nt8yJwWlWfFZEYYKmq9vD3ydkeiDGmJFSVVcknmJmYzOqUEzSqG+o9KXFgDE1q0UmJ5bkH0hdIVtVUVc0GFgIjCuj3HPACcLGQccYBC3wLqnrGuRuCN3SuSDInWO52b2OMMRVJRBjUOZJ3Hu3P4ikD6RvblL98vof457/guaXbOXK6sD9vtZM/AdIGOOBaTnfaLhOR3kC0qi4tYpx7yBcGIrIMOAacxbsX4jYYOKqqe1xtsSKyUURWiMjggh5ERCaJSJKIJGVkZBT1vIwxplC92zVh7v0e/vnvQxjWvSVvrE5j8Atf8PQHW9h7/Hygy6sS/AmQgr7bdnlvQUSCgJeBpwodQKQfkKmqW68YRPVWoBUQDtyYb7Mr9liAw0A7Ve0NPAm8IyINv1eY6hxV9aiqJyoqqsgnZowxxenSIoKX7okj8adDGXtdOz7ceJCbXkxk6jvfsO3Q6UCXF1D+BEg6EO1abgscci1HAD2ARBFJA/oDCb4D6Y6xFPJRlKpeBBJwfSwmIiHAncC7rn5ZqnrCub8BSAG6+FG/McaUWXTTejw3sgcrf34Dk4Z0ZMWuDH74l5U8MG8d6/aeDHR5AeFPgKwHOotIrIiE4Q2DBN9KVT2tqpGqGqOqMcAaYLjvILqzh3IX3mMnOG0NRKSVcz8EuB3Y6XrMm4Gdqpru2ibK9U2tDkBnILUUz9kYY0qteUQdnr6tK6uevpGf/qALW9JPc/crX3PX7NUs33mMmnxqRH7FXjRfVXNEZBqwDAgGXlfVbSLyLJCkqglFj8AQIF1V3X/s6+PdSwl3xvwCcH8tt6A9liHAsyKSA+QCk1W1dsa+MSbgGtUNZdqNnXl4UAfeXb+fOV+m8uAb67m6VUOmDO3I7T1bEVzDp9y1EwmNMaYcZOfksWTTQWatSCE14zwxzbwnJY66tvqdlGhnomMBYoypfLl5yj+3HWFmYgrfHjxNi4bhPDq4A+P6tqN+NZkp0QIECxBjTOCoKl/tOc7MxGTWpJ6kcb1QHhgYwwMDY2hcr2qflGgBggWIMaZq2LDvO2YlJvPZjmPUCwtmfL92PDK4Ay0a1gl0aQWyAMECxBhTtew8cobZiSkkbD5ESJD3OlyTr+9A+2b1A13aFSxAsAAxxlRN+09k8sqXKby/IZ2c3Dx+2Ks1U4Z25OpW3zs3OiAsQLAAMcZUbcfOXOS1lXt5e80+zmfncmPX5kwZ2hFPTNOA1mUBggWIMaZ6OJ15iTe/TmPeqr18l3mJvrFNmTK0I9d3iQrITIkWIFiAGGOql8zsHBauO8Dcr1I5fPoi3Vs3ZMrQTgzr0bJST0q0AMECxBhTPWXn5PHRxoPMXpFC6vHzdIisz+TrOzKydxvCQip+yl0LECxAjDHVW26e8snWI8xMTGbboTO0alSHRwZ3YFzfaOqFVdxJiRYgWIAYY2oGVeXLPceZsTyZdXtP0qReKA/GxzJxQAyN6oWW++NZgGABYoypeTbsO8nM5Sl8vvMY9cOCua9/ex4eFEvzcjwp0QIECxBjTM214/AZZiWmsHTLIUKCgxjTpy2Th3SkXbN6ZR7bAgQLEGNMzZd2/DyvfJnKBxvSycnL445rWvP40I50bVn6kxItQLAAMcbUHkfPXOTVr1KZv3Y/mdm5/LBnK6bf27tU55H4GyDV49rCxhhjitSiYR1+8cNuTL2hE2+u3kd2bm6Fn4To1xeKRWSYiOwSkWQRebqIfmNERH3zoYvIeBHZ5LrliUics+4TEdksIttEZLZrutpfichB1za3u8Z/xqlhl4jcWranbowxNU/jemH85ObO/OzWrhX+WMXugTh/2GcAtwDpwHoRSVDV7fn6RQBPAGt9bao6H5jvrO8JLFHVTc7qu1X1jHgjchFXzpv+sqr+Md/43fBOddsdaA18JiJdVDW3hM/ZGGNMOfBnD6QvkKyqqaqajfeP/IgC+j0HvABcLGSccbjmOVfVM87dECAMKO5gzAhgoapmqepeINmpzRhjTAD4EyBtgAOu5XSn7TIR6Q1Eq+rSIsa5B1eAONstA44BZ/HuhfhME5EtIvK6iDTxtw5jjDGVx58AKegozOW9BREJAl4Gnip0AJF+QKaqbr1iENVbgVZAOHCj0zwL6AjEAYeBF/2pw/VYk0QkSUSSMjIyCivJGGNMGfkTIOlAtGu5LXDItRwB9AASRSQN6A8k+A6kO8aSb+/DR1UvAgk4H4up6lFVzVXVPGAu//qYqrg6fOPNUVWPqnqioqL8eHrGGGNKw58AWQ90FpFYEQnDGwYJvpWqelpVI1U1RlVjgDXAcFVNgst7KO4D5IhIAxFp5dwPAW4HdjrLrVyPPQrw7bUkAGNFJFxEYoHOwLpSPGdjjDHloNhvYalqjohMA5YBwcDrqrpNRJ4FklQ1oegRGAKkq2qqq60+3r2UcGfML4DZzroXnK/6KpAGPObUsU1E3gO2AznAVPsGljHGBI6diW6MMeYKdikTQEQygH1lGCISOF5O5ZQnq6tkrK6SsbpKpibW1V5Viz2IXKMDpKxEJMmfFK5sVlfJWF0lY3WVTG2uq+LnRjTGGFMjWYAYY4wpFQuQos0JdAGFsLpKxuoqGaurZGptXXYMxBhjTKnYHogxxphSqTUBUtycJiLypIhsdy7i+LmItHetmygie5zbRFd7HxH51hnzL1KK2VtKW5eIxInI1858KltE5B7XNm+IyF7XnCpxlVWXsy7X9dgJrvZYEVnrvI7vOlc2qJS6ROQGuXJumosiMtJZVxmv12TnvbJJRFY60xP41hU4z01xY1ZkXSJyi4hscNZtEJEbXdskOmP6Xq/mlVhXjIhccD32bNc2lfH7WFhdRc19VObXy5/aXP2umJfJaauY95iq1vgb3rPdU4AOeC8dvxnolq/PDUA95/7jwLvO/aZAqvOziXO/ibNuHTAA74UePwZuq8S6ugCdnfut8V54srGz/AYwJhCvl7N8rpBx3wPGOvdnA49XZl2uPk2Bk65+lfF6NXTdHw584tzv5vQPB2KdcYL9GbOC6+oNtHbu9wAOuvolAp4AvV4xwNZCxq2M38cC68rXpyeQWl6vl7+1Of0igC/xXlLKU9HvsdqyB1LsnCaqulxVM53FNXgv1ghwK/Cpqp5U1e+AT4Fh4r1mV0NV/Vq9/0p/BUZWVl2qultV9zj3D+G9LH55XT2yLK9XgZz/Dd7Ivy7b/yaV+HrlMwb42NWvrPyp64xrsT7/upJ0YfPc+DsPT4XUpaobnfcVwDagjngvPVQeyvJ6FagSfx/9qeuKuY/KSVnmZaqw91htCZCSziXyMN7/wRS1bRvnvr9jlnddl4lIX7z/g0hxNf/G+Rjn5VL84pe1rjrivaT+Gt/HREAz4JSq5vg5ZkXU5VPQ1aEr/PUSkakikoL3F/yJYrYtj/lvylKX22hgo6pmudrmOR/H/LIUHxWVta5YEdkoIitEZLBrzEr5ffTj9fre3EeU7fXyqzYpfF6mCnuP1ZYA8WsuEQARuQ/wAH8oZlu/x6ygunztrYC3gAfVewl8gGeArsB1eD+u+Xkl19VOvWfA3gv8SUQ6lmTMCqzL93r1xHtxUJ9Keb1UdYaqdnTG/69itq2016uQurwDiHQHnse5qKljvKr2BAY7twmVWNdhvO+v3sCTwDsi0tDfMSuwLu8ABc99VNbXq9japOh5mSrsPVZbAsSvuURE5GbgF3gvR59VzLbpXPnxSIFjVmBdOL84fwf+S1XX+NpV9bB6ZQHzKPnUv2Wqy/fRh3qvwJyI9/P040Bj8V6+v9AxK7Iux93AYlW95Kq3Ul4vl4X86+OVot5fJRmzvOtCRNoCi4H7VfXy3q2qHnR+ngXeoRJfL+djmBPO/Q1497q7UIm/jwXV5fK9vdtyeL38qa2oeZkq7j1WkgMm1fWG97L1qXgPIPkOFnXP16c33jdj53ztTYG9eA+gN3HuN3XWrXf+oXwH7W6vxLrCgM+Bfytg3FbOTwH+BPy+EutqAoQ79yOBPTgH5oD3ufIg+pTKqsu1fg1wQwBer86u+3fgnQoBoDtXHuBMxXtws9gxK7iuxk7/0QWMGencD8V7TGtyJdYVBQQ79zsAB6nc38cC63KWg/D+Ue5Qnq+Xv7Xl65/Ivw6iV9x7rKRPpLre8E5atRvvH5dfOG3P4v1fKsBnwFFgk3NLcG37EN4DT8l4PyrytXvwTniVAkzHOTGzMuoC7gMuudo3AXHOui+Ab53a3gYaVGJdA53H3uz8fNg1Zge835RJxhsm4ZX87xiD9w9OUL4xK+P1+jPeg9GbgOXuX1S8e0spwC5c3xwqaMzKqgvvRzPn872/muM9cLwB2OJs92ecP+iVVNdop30z8A1wRyX/Phb17zgUWJNvvHJ5vfypLV/fRFzf/Kqo95idiW6MMaZUassxEGOMMeXMAsQYY0ypWIAYY4wpFQsQY4wxpWIBYowxplQsQIwxxpSKBYgxxphSsQAxxhhTKv8Pm6Y7tMfCNl4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "train_means = grid_search.cv_results_['mean_train_score']\n",
    "train_stds = grid_search.cv_results_['std_train_score']\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 降低学习率，重新调整n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:42:43.713665Z",
     "start_time": "2020-06-25T08:28:54.623833Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 3713\n",
      "best cv score: 0.465743528446\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':60,\n",
    "          'min_child_samples':40,\n",
    "          'max_bin': 127, \n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params , x_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用所有的训练数据在最优参数的基础上重新训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:50:46.062987Z",
     "start_time": "2020-06-25T08:44:28.692033Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=0.4, importance_type='split', learning_rate=0.01,\n",
       "        max_bin=127, max_depth=7, min_child_samples=40,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=3713,\n",
       "        n_jobs=4, num_class=9, num_leaves=60, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.8, subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "         'objective': 'multiclass',\n",
    "         'num_class': 9,\n",
    "         'n_jobs': 4,\n",
    "         'learning_rate': 0.01,\n",
    "         'n_estimators': n_estimators_2,\n",
    "         'max_depth': 7,\n",
    "         'num_leaves': 60,\n",
    "         'min_child_samples':  40,\n",
    "         'max_bin': 127,\n",
    "         'subsample': 0.8,\n",
    "         'bagging_freq': 1,\n",
    "         'colsample_bytree': 0.4,\n",
    "         }\n",
    "lg = LGBMClassifier(silent = False, **params)\n",
    "lg.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:51:00.086870Z",
     "start_time": "2020-06-25T08:50:54.846357Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle as pk\n",
    "pk.dump(lg, open(\"Otto_LightGBM_org_tfidf.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:50:46.227547Z",
     "start_time": "2020-06-25T08:50:46.224555Z"
    }
   },
   "source": [
    "## 模型测试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:53:45.134522Z",
     "start_time": "2020-06-25T08:53:31.763144Z"
    }
   },
   "outputs": [],
   "source": [
    "test1 = pd.read_csv(\"Otto_FE_test_org.csv\")\n",
    "test2 = pd.read_csv(\"Otto_FE_test_tfidf.csv\")\n",
    "test2 = test2.drop([\"id\"], axis = 1)\n",
    "test = pd.concat([test1, test2], axis = 1, ignore_index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:54:29.724357Z",
     "start_time": "2020-06-25T08:54:29.705378Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1</th>\n",
       "      <th>feat_2</th>\n",
       "      <th>feat_3</th>\n",
       "      <th>feat_4</th>\n",
       "      <th>feat_5</th>\n",
       "      <th>feat_6</th>\n",
       "      <th>feat_7</th>\n",
       "      <th>feat_8</th>\n",
       "      <th>feat_9</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.421803</td>\n",
       "      <td>0.052224</td>\n",
       "      <td>0.842245</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.032787</td>\n",
       "      <td>0.039216</td>\n",
       "      <td>0.21875</td>\n",
       "      <td>0.228571</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.143963</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.070171</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.019608</td>\n",
       "      <td>0.18750</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078248</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.071995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.139311</td>\n",
       "      <td>0.034257</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.026316</td>\n",
       "      <td>0.026316</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.556178</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 187 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id    feat_1    feat_2   feat_3    feat_4  feat_5  feat_6    feat_7  \\\n",
       "0   1  0.000000  0.000000  0.00000  0.000000     0.0     0.0  0.000000   \n",
       "1   2  0.032787  0.039216  0.21875  0.228571     0.0     0.0  0.000000   \n",
       "2   3  0.000000  0.019608  0.18750  0.014286     0.0     0.0  0.000000   \n",
       "3   4  0.000000  0.000000  0.00000  0.014286     0.0     0.0  0.000000   \n",
       "4   5  0.016393  0.000000  0.00000  0.014286     0.0     0.0  0.026316   \n",
       "\n",
       "     feat_8  feat_9      ...        feat_84_tfidf  feat_85_tfidf  \\\n",
       "0  0.000000     0.0      ...                  0.0       0.000000   \n",
       "1  0.000000     0.0      ...                  0.0       0.000000   \n",
       "2  0.000000     0.0      ...                  0.0       0.000000   \n",
       "3  0.000000     0.0      ...                  0.0       0.139311   \n",
       "4  0.026316     0.0      ...                  0.0       0.000000   \n",
       "\n",
       "   feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  feat_90_tfidf  \\\n",
       "0       0.421803       0.052224       0.842245       0.000000            0.0   \n",
       "1       0.000000       0.000000       0.000000       0.143963            0.0   \n",
       "2       0.000000       0.000000       0.078248       0.000000            0.0   \n",
       "3       0.034257       0.000000       0.000000       0.000000            0.0   \n",
       "4       0.000000       0.000000       0.000000       0.000000            0.0   \n",
       "\n",
       "   feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0       0.000000       0.000000       0.000000  \n",
       "1       0.000000       0.070171       0.000000  \n",
       "2       0.000000       0.000000       0.071995  \n",
       "3       0.000000       0.000000       0.000000  \n",
       "4       0.556178       0.000000       0.000000  \n",
       "\n",
       "[5 rows x 187 columns]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:56:58.151092Z",
     "start_time": "2020-06-25T08:56:57.067971Z"
    }
   },
   "outputs": [],
   "source": [
    "test_id = test['id']\n",
    "x_test = test.drop([\"id\"], axis = 1)\n",
    "feat_names = x_test.columns\n",
    "\n",
    "x_test = csr_matrix(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### load模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:04:51.852975Z",
     "start_time": "2020-06-25T08:59:15.723276Z"
    }
   },
   "outputs": [],
   "source": [
    "model = pk.load(open(\"Otto_LightGBM_org_tfidf.pkl\", 'rb'))\n",
    "#输出每类的概率\n",
    "y_test_pred = model.predict_proba(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:05:33.035833Z",
     "start_time": "2020-06-25T09:05:33.031846Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(144368, 9)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_pred.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 生成提交结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:07:29.518083Z",
     "start_time": "2020-06-25T09:07:27.330968Z"
    }
   },
   "outputs": [],
   "source": [
    "out_df = pd.DataFrame(y_test_pred)\n",
    "columns = np.empty(9, dtype = object)\n",
    "for i in range(9):\n",
    "    columns[i] = 'Class_' + str(i + 1)\n",
    "    \n",
    "out_df.columns = columns\n",
    "\n",
    "out_df = pd.concat([test_id, out_df], axis = 1)\n",
    "out_df.to_csv(\"LightGBM_org_tfidf.csv\", index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提交kaggle网站得分如下：<br>\n",
    "![1](http://qckw6brve.bkt.clouddn.com/1.jpg)<br>\n",
    "Private Leaderboard排名第548位<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 采用tfidf特征，采用LightGBM（goss）完成Otto商品分类，尽可能将超参数调到最优，并用该模型对测试数据进行测试，提交Kaggle网站，提交排名。（20分） "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T10:41:21.647446Z",
     "start_time": "2020-06-27T10:41:16.743979Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:20:57.288390Z",
     "start_time": "2020-06-27T11:20:50.688063Z"
    }
   },
   "outputs": [],
   "source": [
    "train1 = pd.read_csv(\"Otto_FE_train_org_goss.csv\")\n",
    "train2 = pd.read_csv(\"Otto_FE_train_tfidf_goss.csv\")\n",
    "\n",
    "#去掉多余的id\n",
    "train2 = train2.drop([\"id\",\"target\"], axis=1)\n",
    "train =  pd.concat([train1, train2], axis = 1, ignore_index=False)\n",
    "train.head()\n",
    "\n",
    "del train1\n",
    "del train2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据准备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:21:01.328173Z",
     "start_time": "2020-06-27T11:21:00.866407Z"
    }
   },
   "outputs": [],
   "source": [
    "# 将类别字符串变成数字，LightGBM不支持字符串格式的特征输入/标签输入\n",
    "y_train = train['target'] #形式为Class_x\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s) - 1)#将类别的形式由Class_x变为0-8之间的整数\n",
    "\n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T10:44:18.544770Z",
     "start_time": "2020-06-27T10:44:18.541803Z"
    }
   },
   "source": [
    "## 参数调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:21:05.189476Z",
     "start_time": "2020-06-27T11:21:05.185487Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:21:08.287193Z",
     "start_time": "2020-06-27T11:21:08.282209Z"
    }
   },
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 100000\n",
    "#直接调用lightgbm内嵌的交叉验证(cv)，可对连续的n_estimators参数进行快速交叉验证\n",
    "#而GridSearchCV只能对有限个参数进行交叉验证，且速度相对较慢\n",
    "def get_n_estimators(params , X_train , y_train , early_stopping_rounds=10):\n",
    "    lgbm_params = params.copy()\n",
    "    lgbm_params['num_class'] = 9\n",
    "     \n",
    "    lgbmtrain = lgbm.Dataset(X_train , y_train )\n",
    "     \n",
    "    #num_boost_round为弱分类器数目，下面的代码参数里因为已经设置了early_stopping_rounds\n",
    "    #即性能未提升的次数超过过早停止设置的数值，则停止训练\n",
    "    cv_result = lgbm.cv(lgbm_params , lgbmtrain , num_boost_round=MAX_ROUNDS , nfold=3,  metrics='multi_logloss' , early_stopping_rounds=early_stopping_rounds,seed=3 )\n",
    "     \n",
    "    print('best n_estimators:' , len(cv_result['multi_logloss-mean']))\n",
    "    print('best cv score:' , cv_result['multi_logloss-mean'][-1])\n",
    "     \n",
    "    return len(cv_result['multi_logloss-mean'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:25:10.594819Z",
     "start_time": "2020-06-27T11:24:18.182994Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 335\n",
      "best cv score: 0.489391399875\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'num_leaves': 60,\n",
    "          'max_depth': 6,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "\n",
    "n_estimators_1 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### num_leaves & max_depth=7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:30:21.341901Z",
     "start_time": "2020-06-27T11:25:27.530492Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  3.4min remaining:  1.7min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  4.9min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=0.7,\n",
       "        importance_type='split', learning_rate=0.1, max_bin=127,\n",
       "        max_depth=7, min_child_samples=20, min_child_weight=0.001,\n",
       "        min_split_gain=0.0, n_estimators=335, n_jobs=4, num_class=9,\n",
       "        num_leaves=31, objective='multiclass', random_state=None,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'num_leaves': range(50, 90, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "num_leaves_s = range(50,90,10) #50,60,70,80\n",
    "tuned_parameters = dict( num_leaves = num_leaves_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4, param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:30:30.214470Z",
     "start_time": "2020-06-27T11:30:30.210479Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.489489762269\n",
      "{'num_leaves': 60}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:30:33.721195Z",
     "start_time": "2020-06-27T11:30:33.647338Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ4AAAELCAYAAADp1+D/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VfW97//XOxNjGBPmIQGCiqggW0DrjFhte8D2Wg84VhHUSh3wnFt7z73n1197e2+Pp4parQoIYh1wbIuetoooDsUAQZkRCHMYwzwPCZ/7x17BLQQSIXuvvZPP8/HYj539Xd+11ufrNvvNd+2VtWRmOOecc4mSFnYBzjnn6hYPHueccwnlweOccy6hPHicc84llAePc865hPLgcc45l1AePM455xLKg8c551xCefA455xLqIywC0hGOTk5lpeXF3YZzjmXUmbPnr3FzHKr6ufBU4m8vDyKiorCLsM551KKpNXV6eeH2pxzziWUB49zzrmE8uBxzjmXUB48zjnnEsqDxznnXEJ58DjnnEsoDx7nnHMJ5cFTgw6WlTNx+irW79gfdinOOZe0PHhq0JY9h/jf/7WIpz4qDrsU55xLWh48Nah9swYM7duJ12etZe22fWGX45xzScmDp4bde0U30tPEE1OXhV2Kc84lJQ+eGta6SX1u6d+Zt78oYUXpnrDLcc65pOPBEwd3X96V+pnpPutxzrlKePDEQU7jetx2UR6T565nycbdYZfjnHNJxYMnTu66tAuNszJ4/IOlYZfinHNJxYMnTpo1zOKOi/P524KNLFi3M+xynHMuaXjwxNGwS/Jp2iCT0VN81uOccxU8eOKoSf1MRlzahalfbebLNdvDLsc555JCXINH0jWSlkgqlvTwSfpdL8kkRYLXWZImSJovaa6ky2P6/kbSWkl7jtnGKEmLJM2TNFVS55hl5ZLmBI/JcRjqCf3kojxaNsriMZ/1OOccEMfgkZQOPA1cC/QAhkrqUUm/bOA+YEZM83AAMzsHGAg8Kqmi1neAvpXs8ksgYmbnAm8Cj8Qs229mvYLHoNMb2bfTqF4G91zelU+XbWHmym2J3LVzziWleM54+gLFZrbCzA4Bk4DBlfT7NdGQOBDT1gOYCmBmm4EdQCR4XWhmG47diJl9ZGYV16kpBDrU1EBO1839O9Mqux6/e38JZhZ2Oc45F6p4Bk97YG3M65Kg7ShJvYGOZvbuMevOBQZLypCUD/QBOn6LfQ8D/hbzur6kIkmFkq77FtupEfUz07n3im7MXLmNfxRvTfTunXMuqcQzeFRJ29F/7geHzkYDD1XSbzzRoCoCHgemA2XV2ql0M9HZ0X/GNHcyswhwI/C4pK6VrDciCKei0tLS6uzqWxnStyPtmtbn0Sk+63HO1W3xDJ4SvjlL6QCsj3mdDfQEpklaBfQHJkuKmFmZmT0YfCczGGgGVHn9GUlXAf8GDDKzgxXtZrY+eF4BTAN6H7uumY0xs4iZRXJzc7/dSKuhXkY6PxtQwJdrdvDRks01vn3nnEsV8QyeWUCBpHxJWcAQ4OgZZWa208xyzCzPzPKIfi8zyMyKJDWU1AhA0kCgzMwWnWxnwWG754JtbI5pby6pXvBzDvAd4KTbipfr+3SgU4uGPDZlqc96nHN1VtyCx8zKgJHAe8Bi4HUzWyjpV5KqOrOsFfCFpMXAz4FbKhZIekRSCdBQUomkXwaL/hNoDLxxzGnTZwFFkuYCHwG/rSrE4iUzPY37BxSwYN0u3lu4KYwSnHMudPJ/eR8vEolYUVFRXLZdfsQYOPpjMtPS+Nv9l5CWVtlXYc45l3okzQ6+Tz8pv3JBgqWniQev6s6STbt5d/5xZ4U751yt58ETgu+f05Yz22Tz+JSllJUfCbsc55xLKA+eEKSliQcHdmfFlr38ec76qldwzrlaxIMnJFf3aM057ZvyxNSlHPZZj3OuDvHgCYkkRg3sztpt+3mjqCTscpxzLmE8eEJ0+Rm5nN+pGb//cBkHDpeHXY5zziWEB0+IJPHQ1WewYecBJs1cE3Y5zjmXEB48Ibuoa0v6d2nB09OWs/+Qz3qcc7WfB0/IKmY9pbsP8lLh6rDLcc65uPPgSQIX5LXgkoIcnvl4OXsOVusi3M45l7I8eJLEQ1efwba9h5g4fVXYpTjnXFx58CSJXh2bcdVZrXju4+Xs3H847HKccy5uPHiSyIMDu7PrQBnPf7Yy7FKccy5uPHiSyNntmvK9c9ow/rOVbN97KOxynHMuLjx4kswDV3Vn76EynvtkRdilOOdcXHjwJJnurbMZfF47Jk5fRenug1Wv4JxzKcaDJwndf1V3DpUf4dmPl4ddinPO1bi4Bo+kayQtkVQs6eGT9LtekkmKBK+zJE2QNF/SXEmXx/T9jaS1kvYcs416kl4L9jVDUl7Msl8E7UskfbfGB1rD8nMa8aPe7flj4Wo27jwQdjnOOVej4hY8ktKBp4FrgR7AUEk9KumXDdwHzIhpHg5gZucAA4FHJVXU+g7Qt5JdDgO2m1k3YDTwH8H2ewBDgLOBa4A/BLUltfsGFGBmPP1RcdilOOdcjYrnjKcvUGxmK8zsEDAJGFxJv18DjwCx/7TvAUwFMLPNwA4gErwuNLPK7hk9GJgY/PwmMECSgvZJZnbQzFYCxVQeXEmlY4uG3BDpyKRZayjZvi/scpxzrsbEM3jaA2tjXpcEbUdJ6g10NLN3j1l3LjBYUoakfKAP0LG6+zOzMmAn0LI6dSSrkVd2QxK/n+qzHudc7RHP4FElbXZ0YfTQ2WjgoUr6jScaEEXA48B0oKqLmJ1ofyetI6aeEZKKJBWVlpZWsavEaNu0ATf168SbX5SwasvesMtxzrkaEc/gKeGbs5QOwPqY19lAT2CapFVAf2CypIiZlZnZg2bWy8wGA82AZdXdn6QMoCmwrRp1AGBmY8wsYmaR3NzcbzHM+Lrn8q5kposnplY1fOecSw3xDJ5ZQIGkfElZRL/gn1yx0Mx2mlmOmeWZWR5QCAwysyJJDSU1ApA0ECgzs0VV7G8ycFvw8/XAh2ZmQfuQ4Ky3fKAAmFmD44yrVtn1ue2iPP48Zx3LNu0OuxznnDttcQue4HuWkcB7wGLgdTNbKOlXkgZVsXor4AtJi4GfA7dULJD0iKQSoKGkEkm/DBY9D7SUVAyMAh4O6lgIvA4sAv4O3GtmKXXHtbsu7UrDzHQe/8BnPc651KfopMDFikQiVlRUFHYZ3/DY+0t48sNi/nrfJfRo1yTscpxz7jiSZptZpKp+fuWCFDHski40qZ/B6A+Whl2Kc86dFg+eFNG0QSbDL+nClEWbmLt2R9jlOOfcKfPgSSG3X5xP84aZPDbFZz3OudTlwZNCGtfL4O7LuvLx0lKKVm0LuxznnDslHjwp5tYL88hpXI9H3/dZj3MuNXnwpJgGWence0VXPl+xlenFW8IuxznnvjUPnhQ0tG8n2jatz6NTluKnwzvnUo0HTwqqn5nOyCu7MXv1dj5emhzXlXPOuery4ElRP+7TkQ7NG/CYz3qccynGgydFZWWkcd+AAuaV7GTKok1hl+Occ9XmwZPCftS7Pfk5jXhsylKOHPFZj3MuNXjwpLCM9DQeuKqArzbu5q8LKrspq3POJR8PnhT3g3Pb0b11Yx7/YBnlPutxzqUAD54Ul54mHryqO8Wb9zB57rqwy3HOuSp58NQC3z27DT3aNuHxD5ZxuPxI2OU459xJefDUAmlp4qGru7N66z7e/qIk7HKcc+6kPHhqiSvPbEWvjs14cmoxB8tS6garzrk6Jq7BI+kaSUskFUt6+CT9rpdkkiLB6yxJEyTNlzRX0uUxffsE7cWSnpSkoP01SXOCxypJc4L2PEn7Y5Y9G88xh0WKznrW7djP67PWhl2Oc86dUNyCR1I68DRwLdADGCqpRyX9soH7gBkxzcMBzOwcYCDwqKSKWp8BRgAFweOaoO8/m1kvM+sFvAW8HbO95RXLzOzuGhxmUrm4Ww5981rw1EfFHDjssx7nXHKK54ynL1BsZivM7BAwCRhcSb9fA48AB2LaegBTAcxsM7ADiEhqCzQxs88tep2YF4HrYjcWzIBuAF6t4fEkPUmMuro7m3Yd5KXC1WGX45xzlYpn8LQHYo/5lARtR0nqDXQ0s3ePWXcuMFhShqR8oA/QMVg/9tvz47YJXAJsMrNlMW35kr6U9LGkS055RCmgf5eWXNwth2c/Xs7eg2Vhl+Occ8eJZ/Cokrajf+EYHDobDTxUSb/xREOlCHgcmA6UVbXNwFC+OdvZAHQys97AKOAVSU2OK1YaIalIUlFpaWpf8XnU1d3ZsucQEz9fFXYpzjl3nHgGTwnRWUqFDsD6mNfZQE9gmqRVQH9gsqSImZWZ2YPBdzKDgWbAsmCbHU60TUkZwI+A1yrazOygmW0Nfp4NLAe6H1usmY0xs4iZRXJzc09j2OE7v1NzrjyzFWM+WcHuA4fDLsc5574hnsEzCyiQlC8pCxgCTK5YaGY7zSzHzPLMLA8oBAaZWZGkhpIaAUgaCJSZ2SIz2wDsltQ/+C7nVuAvMfu8CvjKzI4ejpOUG5zogKQuRE9IWBHHcSeFUQO7s2PfYcZ/tirsUpxz7hviFjxmVgaMBN4DFgOvm9lCSb+SNKiK1VsBX0haDPwcuCVm2T3AOKCY6OzlbzHLhnD8SQWXAvMkzQXeBO42s22nOKyU0bN9U757dmvGfbqCHfsOhV2Oc84dJb+J2PEikYgVFRWFXcZp+2rjLq594lN+enlX/vW7Z4ZdjnOulpM028wiVfXzKxfUYme2acIPzm3HhH+sYuueg2GX45xzgAdPrffAVQUcOFzOsx8vD7sU55wDPHhqva65jflh7w68+PlqNu86UPUKzjkXZx48dcD9AwooP2I8/VFx2KU455wHT13QqWVDfhzpyKsz17Jux/6wy3HO1XEePHXEz67sBsBTHy6roqdzzsWXB08d0a5ZA4b27cgbRSWs2bov7HKcc3WYB08dcu8V3UhPE09M9VmPcy48VQaPpK6S6gU/Xy7pPknN4l+aq2mtmtTn1gs786cvSyjevCfscpxzdVR1ZjxvAeWSugHPA/nAK3GtysXN3Zd1pX5mus96nHOhqU7wHAmuu/ZD4HEzexBoG9+yXLy0bFyP27+Tx7vz1vPVxl1hl+Ocq4OqEzyHJQ0FbgMqbtiWGb+SXLwNv6QLjbMyGD1ladilOOfqoOoEz+3AhcBvzGxlcEfQl+JblounZg2zuPOSLry3cBPzS3aGXY5zro6pMniC++DcZ2avSmoOZJvZbxNQm4ujOy7Oo1nDTB6bsiTsUpxzdUx1zmqbJqmJpBbAXGCCpMfiX5qLp+z6mdx1aVc+WlLK7NXbwy7HOVeHVOdQW1Mz20X0ltITzKwP0Tt9uhR320WdyWmc5d/1OOcSqjrBkyGpLXADX59c4GqBhlkZ3H1ZVz4r3kLhiq1hl+OcqyOqEzy/Inr76uVmNktSF8D/CKSWuLl/Z1o3qcdj7y/F70brnEuE6pxc8IaZnWtm9wSvV5jZf6vOxiVdI2mJpGJJD5+k3/WSTFIkeJ0laYKk+ZLmSro8pm+foL1Y0pOSFLT/UtI6SXOCx/di1vlF0H+JpO9Wp/a6on5mOiOv6MbMVdv4dNmWsMtxztUB1Tm5oIOkP0naLGmTpLckdajGeunA08C1QA9gqKQelfTLBu4DZsQ0Dwcws3OAgcCjkipqfQYYARQEj2ti1httZr2Cx1+D7fcAhgBnB33/ENTmAjdc0JH2zRrw6BSf9Tjn4q86h9omAJOBdkB74J2grSp9geJghnQImAQMrqTfr4FHgNjbY/YApgKY2WZgBxAJvmtqYmafW/QT8kXguirqGAxMMrODZrYSKA5qc4F6GencN6Abc9fu4MOvNoddjnOulqtO8OSa2QQzKwseLwC51VivPbA25nVJ0HaUpN5ARzM79qSFucBgSRnBH6z2AToG65ecZJsjJc2TND74m6Nq1RHUMkJSkaSi0tLSagyvdvnR+R3o3LIhj76/lCNHfNbjnIuf6gTPFkk3S0oPHjcD1TkFSpW0Hf1ECw6djQYeqqTfeKIBUQQ8DkwHyqrY5jNAV6AXsAF4tDp1HG0wG2NmETOL5OZWJ1drl8z0NB64qoBFG3bx3sKNYZfjnKvFMqrR5w7gKaIhYURD4PZqrFdCdJZSoQOwPuZ1NtATmBacH9AGmCxpkJkVAQ9WdJQ0neiZdNuD7Ry3TTPbFNN/LF+f+l1VHS4w6Lz2PP3Rch6bspSrz25Delplme2ciwcz43C5UXbkCIfLjMNHjlBWbhwuP0LZkejz4fJoW9mRIxwqiz4f38coKz/C4SPG4bIj0e2V29frVWzjG33s6LY75zTkF9eeFdexVhk8ZrYGGBTbJukBojORk5kFFASHytYR/YL/xpjt7gRyYrY5DfgXMyuS1BCQme2VNBAoM7NFQb/dkvoTPRnhVuD3QXtbM9sQbO6HwILg58nAK8HVFtoRPSFhZlXjrovS08QDVxUw8pUveXfeegb3Ou6IpHNJKVk+tKN9osvLyqN1VCw7up+y6Lrf7GOUJ+AQtxQ9upGZJjLS08hMF5npaWSki8y06HODrPife1WdGU9lRlFF8JhZmaSRRP8GKB0Yb2YLJf0KKDKzySdZvRXwnqQjREPrlphl9wAvAA2AvwUPgEck9SI6K1sF3BXUsVDS68Aioofr7jWz8m8x1jrlez3bcmabYh7/YBnfP6ctGel+k1pX83buP8zvpy5j695D3+pDu6zcgg//kD6006If1hUf2hlpaWRmfP2hHX1d8cEuGmdmkJEW/XCv+IDPSEsjKyP6nJFesSxYN9h2RprIykiL6RMExDH7zwz6Hg2P9LSva0kXWelpR7eXmZ6WNEcxdCqnz0paa2Ydq+6ZmiKRiBUVFYVdRmjeX7iREX+czSPXn8sNkVr7NruQ7Nx3mFvGz2DR+l20aVo/+HDUCT9UM9LTjusT+0H7dR9940P26z4V68Vsv5J/8WekxewnmAFkZiTfh3YykzTbzCJV9TvVGY+f9lSLDezRmnM7NOXJqcu4rld7sjJ81uNqxs59h7n5+Rl8tXEXz97ch6t6tA67JBeCE36iBN+l7KrksZvodyWulpLEqIHdKdm+n9eL1la9gnPVsGPfIW56vpAlG3d76NRxJwweM8s2syaVPLLN7FRnSi5FXNY9lz6dm/PUh8UcOOxfibnTs2PfIW4aN4OlG/fw3C19GHCWh05d5sdQXKUk8dDV3dm46wCvzlwTdjkuhW3fe4gbx85g2eY9PHdrH644s1XYJbmQefC4E7qoaw4XdmnJ0x8tZ/8hn/W4b2/b3kPcOG4GxaV7GHNLH644w0PHefC4Kjx0dXe27DnIi5+vCrsUl2K27T3EjWMLWV66h7G3RrjcQ8cFPHjcSUXyWnBZ91ye/Xg5ew6WhV2OSxFb9xzkxrGFrNyyl3G3Rrise927DJU7sercFqGys9vWBrdK6JKIIl24Rg3szvZ9h5nw2cqwS3EpYOueg9w0bkY0dG6LcKmHjjtGdWY8jwH/SvSKzh2AfwHGEr3Nwfj4leaSxXkdmzGwR2vGfLqCnfsOh12OS2Jb9hzkxrHR0Hn+tgu4pMBDxx2vOsFzjZk9Z2a7zWyXmY0BvmdmrwHNq1rZ1Q6jBnZn94Eyxn22IuxSXJLaEhxeW71tL+N/cgEXF+RUvZKrk6oTPEck3SApLXjcELPMr2BQR5zVtgnfP7ct4z9byba9h8IuxyWZ0t0HGTqmkDXb9jH+tgv4TjcPHXdi1Qmem4hepHNz8LgFuFlSA2BkHGtzSebBqwrYf7ic5z5ZHnYpLols3n2AoWMLWbt9H+N/cgEXeei4KlQZPMGtq//JzHKCxz+ZWbGZ7TezzxJRpEsO3VplM7hXeyZOX8Xm3QeqXsHVept3HWDomELWbd/PhJ/05aKuHjquatU5q61DcAbbZkmbJL0lqUNV67na6f4BBRwuN56Z5rOeum7zrgMMGVvI+h0HmHD7BVzYtWXYJbkUUZ1DbROI3kytHdEz294J2lwdlJfTiOvP78DLhWvYsHN/2OW4kGzadYAhYwrZuPMAL9x+Af27eOi46qtO8OSa2QQzKwseLwB+jmQd9rMB3TCMpz4sDrsUF4JNweG1jbsO8MLtfennoeO+peoEzxZJN0tKDx43A1vjXZhLXh2aN2TIBZ14vWgta7ftC7scl0Abd0ZnOpt2HWDiHX3pm98i7JJcCqpO8NwB3ABsBDYA1wO3V2fjkq6RtERSsaSHT9LvekkmKRK8zpI0QdJ8SXMlXR7Tt0/QXizpSUkK2v9T0leS5gXfSTUL2vMk7Zc0J3g8W53a3cnde0U3JPHk1GVhl+ISZMPO/QwZ8zmbg9C5IM9Dx52a6pzVtsbMBplZrpm1MrPrgB9VtZ6kdOBp4FqgBzBUUo9K+mUD9wEzYpqHB/s+BxgIPCqpotZngBFAQfC4JmifAvQ0s3OBpcAvYra33Mx6BY+7q6rdVa1N0/rc0r8zb3+5jhWle8Iux8XZ+h37GTKmkC17DvHisL5EPHTcaTjVi4SOqkafvkBxcDr2IaKX2BlcSb9fA48Asefn9gCmApjZZmAHEJHUFmhiZp+bmQEvAtcF/d43s4qrWBYSvbyPi6N7Lu9KVnoaT/isp1arCJ2tew4x8Y6+9OnsoeNOz6kGj6rRpz0Qe9/kkqDt641IvYGOZvbuMevOBQZLypCUD/QBOgbrl5xsm4E7gL/FvM6X9KWkjyVdUumApBGSiiQVlZaWVmN4LqdxPW67KI/Jc9ezdNPusMtxcbAuCJ3te6MznT6d/SpZ7vSdavBU51I5lYXT0fWCQ2ejgYcq6TeeaKgUAY8D04GyqrYZbPffgr4vB00bgE5m1pvoTO0VSU2O24jZGDOLmFkkN9dP2quuuy7tQqOsDB7/YGnYpbgaVrJ9H0PGfH40dM7v5KHjakbGiRZI2k3lASOgQTW2XUJ0llKhA7A+5nU20BOYFpwf0AaYLGmQmRUBD8bUMh1YBmznm4fQvrFNSbcBPwAGBIfiMLODwMHg59mSlgPdiYaaO03NG2Vxx8X5PDl1GQvX7+Tsdk3DLsnVgLXb9jF0bCE79x/mj3f2o1fHZmGX5GqRE854zCzbzJpU8sg2sxMGVoxZQIGkfElZwBCif4hasf2dwSV48swsj+j3MoPMrEhSQ0mNACQNBMrMbJGZbQB2S+ofnM12K/CXoN81wM+DbRw9x1dSbnCiA8H9gwoAv8RyDRp2cT5NG2QyeorPemqDtdv2MWRMNHReGuah42pe3O5AGnzRPxJ4D1gMvG5mCyX9StKgKlZvBXwhaTHRMLklZtk9wDigGFjO19/lPEV0FjXlmNOmLwXmSZoLvAncbWbbTn+ErkLTBpmMuLQLHyzezJy1O8Iux52GitDZfeAwL9/Zj/M8dFwcKDgi5WJEIhErKvIjcd/G3oNlXPLIR5zdrgl/HNYv7HLcKVizNXp4bc/BMl4a1o9zOvhhU/ftSJptZpGq+sVtxuPqlkb1Mrjnsq58umwLM1f6hDLVrNkaPZFgz8EyXr7TQ8fFlwePqzE39+9MbnY9Hn1/CT6TTh2rt+5lyJjP2Xe4nJfv7EfP9h46Lr48eFyNaZCVzsgrujFj5TamL/fL+aWCVVv2MmRMoYeOSygPHlejhvTtSLum9X3WkwIqQufA4XJeubO/nwrvEsaDx9WoehnpjLyygC/W7GDaEr8CRLJauWUv/zzmcw6VH+GV4f3p0e64v6l2Lm48eFyN+3GkA51aNOTRKT7rSUYrSvcwZMznHC43Xhnej7Paeui4xPLgcTUuMz2N+wYUsGDdLt5buCnsclyM5aV7GDKmkLJy49Xh/TmzjYeOSzwPHhcX1/VqR5fcRoyespQjR3zWkwyKN+9h6JhCyo8YrwzvzxltssMuydVRHjwuLjLS03jgqu4s2bSb/5q/Iexy6rzizXsYOraQI2a8OsJDx4XLg8fFzQ/OacsZrbMZ/cFSysqPhF1OnVW8eTdDxhRiBq8O70/31h46LlwePC5u0tLEgwO7s6J0L3+Zs77qFVyNW7YpGjoAk0b0o8BDxyUBDx4XV989uzU92zfhianLOOyznoRaumk3Q8cWIolJI/rTrZWHjksOHjwuriQxamB31mzbx5uzS6pewdWIJRt3M3RMNHReHd6fbq0ah12Sc0d58Li4u+KMVvTu1IzfT13GwbLysMup9b7auIsbxxaSnlYx0/HQccnFg8fFnSQeGngG63ceYNLMtWGXU6st3rCLG8fOICM9Gjpdcz10XPLx4HEJ8Z1uLemX34KnPipm/yGf9cTDovXRmU5WehqTRlxIFw8dl6Q8eFxCSOKhq8+gdPdBXipcHXY5tc6i9bu4aVwh9TLSmTSiP/k5jcIuybkTimvwSLpG0hJJxZIePkm/6yWZpEjwOkvSBEnzJc2VdHlM3z5Be7GkJyUpaG8haYqkZcFz86BdQb9iSfMknR/PMbsT65vfgksKcnjm4+XsPVgWdjm1xsL1O7lxXCH1M6Ohk+eh45Jc3IJHUjrwNHAt0AMYKqlHJf2ygfuAGTHNwwHM7BxgIPCopIpanwFGAAXB45qg/WFgqpkVAFOD1wT7r+g7IljfhWTUwO5s23uIF6avCruUWmHBup3cOHYGDT10XAqJ54ynL1BsZivM7BAwCRhcSb9fA48AB2LaehAND8xsM7ADiEhqCzQxs88tetnjF4HrgnUGAxODnyce0/6iRRUCzYLtuBD07tScAWe2YswnK9h14HDY5aS0Bet2ctO4GTSul8GkERfSuaWHjksN8Qye9kDsKUwlQdtRknoDHc3s3WPWnQsMlpQhKR/oA3QM1o/9Y5DYbbY2sw0AwXOr6tbhEuvBgd3Zuf8wz3+6MuxSUtb8kp3cOLYwCJ3+dGrZMOySnKu2eAaPKmk7epni4NDZaOChSvqNJxoQRcDjwHSgrKptnkodMfWMkFQkqai01G9gFk892zfl2p5tGP/ZSrbvPRR2OSlnXskObhpXSHb9TCaN6E/HFh46LrXEM3hKiM5SKnQAYi/YlQ30BKZJWgX0ByZLiphZmZk9aGYLgsgJAAATzUlEQVS9zGww0AxYFmyzwwm2uaniEFrwvLmadQBgZmPMLGJmkdzc3FMasKu+Bwd2Z8+hMsZ8uiLsUlLK3LU7uHncDJo08NBxqSuewTMLKJCULykLGAJMrlhoZjvNLMfM8swsDygEBplZkaSGkhoBSBoIlJnZouAQ2m5J/YOz2W4F/hJscjJwW/Dzbce03xqc3dYf2FlxSM6Fp3vrbAad144X/rGKLXsOhl1OSpizdgc3P++h41Jf3ILHzMqAkcB7wGLgdTNbKOlXkgZVsXor4AtJi4GfA7fELLsHGAcUA8uBvwXtvwUGSlpG9Ey43wbtfwVWBP3HAj893bG5mnH/gAIOlpXzzLTlYZeS9L5cs51bxs2gWcNMXrvrQjo099BxqUvRk8NcrEgkYkVFRWGXUSf86xtzmTx3PZ/89yto3aR+2OUkpS/WbOe252fSvFEWr47oT/tmDcIuyblKSZptZpGq+vmVC1yo7htQQPkR4+mPisMuJSnNXr2dW5+fSYvGWUzy0HG1hAePC1XHFg254YKOvDpzDSXb94VdTlKZvXobt42fScsgdNp56LhawoPHhe5nV3ZDEk996LOeCkWrtnHr8zPJaZzFayMupG1TDx1Xe3jwuNC1bdqAG/t24o3ZJazasjfsckI3a1V0ptOqSX0mjbiQNk39uy9Xu3jwuKTw0yu6kpkunpy6LOxSQjVzZTR0Wjepz6QR/T10XK3kweOSQqvs+tx2YR5/nrOO4s27wy4nFDNWbOUnE2bSpml9Xh3R38/yc7WWB49LGndd1pUGmemM/qDuzXoKV2zl9hdm0bZpfSYN99BxtZsHj0saLRplccfF+fzXvA0s3rAr7HIS5vPlW7l9QjR0Xh3Rn1YeOq6W8+BxSeXOi7uQXT+Dx6YsDbuUhJi+fAt3vDCL9s0bREMn20PH1X4ePC6pNG2YyfBLujBl0SbmlewIu5y4ml4cDZ0OzRvw6nAPHVd3ePC4pHP7d/Jo3jCzVs96/lG8hTsmzqJTi4a8Mrw/udn1wi7JuYTx4HFJJ7t+Jndd1pVpS0qZvXpb2OXUuM+WRWc6nVs08tBxdZIHj0tKt17YmZzG9Xj0/do16/l0WSnDJs4iP6cRrwzvR05jDx1X93jwuKTUMCuDn17elenLtzJ9+Zawy6kRnywtZdjEIvJzGvHynf1o6aHj6igPHpe0buzXiTZN6vPY+0tJ9dt3fLy0lDtfLKJLTvTwmoeOq8s8eFzSqp+Zzsgru1G0ejufLEvdWc+0JZsZ/mIRXXMb8+rw/rRolBV2Sc6FyoPHJbUbIh3p0LwBj76/JCVnPR99tZkRL86mW25jXrmzH809dJyLb/BIukbSEknFkh4+Sb/rJZmkSPA6U9JESfMlLZb0i5i+90taIGmhpAdi2l+TNCd4rJI0J2jPk7Q/Ztmz8Ryzq1lZGWncN6CAeSU7+WDx5rDL+VY+/GoTd/1xNgWtG/PKcA8d5ypkxGvDktKBp4GBQAkwS9JkM1t0TL9s4D5gRkzzj4F6ZnaOpIbAIkmvAo2B4UBf4BDwd0n/ZWbLzOyfY7b5KLAzZnvLzaxXzY/SJcKPerfnmWnLeWzKUgac2Yq0NIVdUpWmLt7EPS99Qfc2jXlpWD+aNfTQca5CPGc8fYFiM1thZoeAScDgSvr9GngEOBDTZkAjSRlAA6Ihsws4Cyg0s31mVgZ8DPwwdmOSBNwAvFrD43EhyUhP4/4BBSzesIu/LdgYdjlV+mDRJu5+aTZntMnm5WH9PXScO0Y8g6c9sDbmdUnQdpSk3kBHM3v3mHXfBPYCG4A1wO/MbBuwALhUUstgJvQ9oOMx614CbDKz2Esc50v6UtLHki453YG5xPun89pR0Koxoz9YSvmR5P2uZ8qiTdzz8mzOatuEl4b1o2nDzLBLci7pxDN4KjsecvQTQ1IaMBp4qJJ+fYFyoB2QDzwkqYuZLQb+A5gC/B2YC5Qds+5Qvjnb2QB0MrPewCjgFUlNjitWGiGpSFJRaWlpNYfoEiU9TTw4sDvFm/cwee66sMup1PsLN/LTl2fTo20T/uih49wJxTN4SvjmbKQDsD7mdTbQE5gmaRXQH5gcnGBwI/B3MztsZpuBfwARADN73szON7NLgW3A0ZlNcGjuR8BrFW1mdtDMtgY/zwaWA92PLdbMxphZxMwiubm5pz14V/OuObsNPdo24YkPllFWfiTscr7hvYUb+enLX9CjXVNeHNaPpg08dJw7kXgGzyygQFK+pCxgCDC5YqGZ7TSzHDPLM7M8oBAYZGZFRA+vXamoRkRD6SsASa2C505EQyZ2dnMV8JWZlVQ0SMoNTnRAUhegAFgRr0G7+ElLE6MGdmfV1n28/UXyzHr+vmAj9778BT3bN+WPw/p66DhXhbgFT/Dl/0jgPWAx8LqZLZT0K0mDqlj9aaJnsC0gGmATzGxesOwtSYuAd4B7zWx7zHpDOP6kgkuBeZLmEv3u6O7g+yKXggac1YrzOjbjianLOFQW/qznb/M3MPKVLzinQ1NeHNaXJvU9dJyrilLxj/LiLRKJWFFRUdhluBP4ZGkpt46fya+v68kt/TuHVsdf52/gZ69+yXkdmjLxjr5ke+i4Ok7SbDOLVNXPr1zgUs4lBTlckNecpz5cxoHD5aHU8F/zoqHTq2MzDx3nviUPHpdyJDFq4Bls2nWQl2esSfj+3523nvsmfUlvDx3nTokHj0tJF3ZtyXe6teSZacXsO3TsGfXx887c9dw/aQ7nd2rGC3f0pXG9uF38w7lay4PHpaxRA89gy55DTJy+OiH7+8ucddw/6Uv6dGrOC7d76Dh3qjx4XMrq07k5V5yRy3OfLGf3gcNx3ddf5qzjwdfmEMlrwYTbL6CRh45zp8yDx6W0UQPPYMe+w4z/bFXc9vHnL6Ohc0FeC17w0HHutHnwuJR2ToemXN2jNeM+W8HOfTU/6/nTlyWMen0OffOjM52GWR46zp0uDx6X8kZd3Z09B8sY+2nNXpDirdkljHp9Lv3yWzL+Jx46ztUUDx6X8s5s04Tvn9OW8f9YydY9B2tkm2/OLuFf3pzLhV08dJyraR48rlZ44KruHDhcznOfnP6s542itfzrm3O5qGtLnr/tAhpkpddAhc65Ch48rlbo1qox1/Vuz8Tpq9i860DVK5zA67PW8t/fmsd3uuZ46DgXJx48rta4f0ABZUeMP0xbfkrrvzZrDT9/ex4Xd8th3G0R6md66DgXDx48rtbo3LIRN0Q68MqMNazfsf9brTtp5hp+/tZ8LinIZeytHjrOxZMHj6tVRl5ZAMDvPyyu9jqvzFjDw2/P57LuuYy5pY+HjnNx5sHjapX2zRowtG9H3ihay5qt+6rs//KM1fyPP83n8jNyec5Dx7mE8OBxtc69V3QjPU08+eGyk/Z7qXA1//anBVxxRi7P3uyh41yiePC4WqdVk/rc0r8zb39RwvLSPZX2+WPhav7nnxdw5ZmteNZnOs4lVFyDR9I1kpZIKpb08En6XS/JJEWC15mSJkqaL2mxpF/E9L1f0gJJCyU9ENP+S0nrJM0JHt+LWfaLoIYlkr4br/G65HH35V2pn5nOEx8cP+t58fNV/K8/L2DAma145ubzqZfhoeNcIsUteCSlA08D1wI9gKGSelTSLxu4D5gR0/xjoJ6ZnQP0Ae6SlCepJzAc6AucB/xAUkHMeqPNrFfw+Guw/R7AEOBs4BrgD0FtrhbLaVyPn1yUxzvz1rNk4+6j7ROnr+Lf/7KQq85qxR88dJwLRTxnPH2BYjNbYWaHgEnA4Er6/Rp4BIj9qz8DGknKABoAh4BdwFlAoZntM7My4GPgh1XUMRiYZGYHzWwlUBzU5mq5EZd2oXFWBqOnLAXghX+s5P+bvJCBPVrzh5v6eOg4F5J4Bk97YG3M65Kg7ShJvYGOZvbuMeu+CewFNgBrgN+Z2TZgAXCppJaSGgLfAzrGrDdS0jxJ4yU1r24drnZq1jCLYZfk8/eFG/n3vyzgl+8s4uoerXn6xvPJyvCvN50LSzx/+1RJmx1dKKUBo4GHKunXFygH2gH5wEOSupjZYuA/gCnA34G5QMV9j58BugK9iAbWo9WpI6aeEZKKJBWVlpZWPTqXEu64OJ+mDTJ58fPVfPfs1jzloeNc6OL5G1jCN2cjHYD1Ma+zgZ7ANEmrgP7A5OAEgxuBv5vZYTPbDPwDiACY2fNmdr6ZXQpsA5YF7ZvMrNzMjgBj+fpwWlV1EKw/xswiZhbJzc09zaG7ZNGkfib/54fncOfF+R46ziWJeP4WzgIKJOVLyiL6Bf/kioVmttPMcswsz8zygEJgkJkVET28dqWiGhENpa8AJLUKnjsBPwJeDV63jdn3D4keliPY5xBJ9STlAwXAzHgN2iWf75/blv/5gx5kpnvoOJcM4naTETMrkzQSeA9IB8ab2UJJvwKKzGzySVZ/GphANDwETDCzecGytyS1BA4D95rZ9qD9EUm9iB5GWwXcFdSxUNLrwCKih+XuNbPymhyrc8656pPZcV931HmRSMSKiorCLsM551KKpNlmFqmqnx97cM45l1AePM455xLKg8c551xCefA455xLKA8e55xzCeXB45xzLqH8dOpKSCoFVp/GJnKALTVUTphqyzjAx5KsastYass44PTG0tnMqrz0iwdPHEgqqs657MmutowDfCzJqraMpbaMAxIzFj/U5pxzLqE8eJxzziWUB098jAm7gBpSW8YBPpZkVVvGUlvGAQkYi3/H45xzLqF8xuOccy6hPHhOk6RVkuZLmiOpKGhrIWmKpGXBc/OqtpMMTjCWX0paF7TNkfS9sOusDknNJL0p6StJiyVdmIrvywnGkXLviaQzYuqdI2mXpAdS9D050VhS8X15UNJCSQskvSqpfnAPtRnBe/JacD+1mt2vH2o7PcHdUyNmtiWm7RFgm5n9VtLDQHMz+3lYNVbXCcbyS2CPmf0urLpOhaSJwKdmNi74xWkI/A9S7H05wTgeIAXfkwqS0oF1QD/gXlLsPYl1zFhuJ4XeF0ntgc+AHma2P7hv2V+B7wFvm9kkSc8Cc83smZrct8944mMwMDH4eSJwXYi11DmSmgCXAs8DmNkhM9tBir0vJxlHqhsALDez1aTYe1KJ2LGkogyggaQMov+o2QBcCbwZLI/Le+LBc/oMeF/SbEkjgrbWZrYBIHhuFVp1305lYwEYKWmepPGpcCgE6AKUAhMkfSlpXHAL9VR7X040Dki99yTWEIJb1pN678mxYscCKfS+mNk64HfAGqKBsxOYDewws7KgWwnQvqb37cFz+r5jZucD1wL3Sro07IJOQ2VjeQboCvQi+j/noyHWV10ZwPnAM2bWG9gLPBxuSafkRONIxfcEgOBw4SDgjbBrOV2VjCWl3pcgGAcD+UA7oBHR3/1j1fj3MR48p8nM1gfPm4E/AX2BTZLaAgTPm8OrsPoqG4uZbTKzcjM7AowlOr5kVwKUmNmM4PWbRD/AU+19qXQcKfqeVLgW+MLMNgWvU+09ifWNsaTg+3IVsNLMSs3sMPA2cBHQLDj0BtABWF/TO/bgOQ2SGknKrvgZuBpYAEwGbgu63Qb8JZwKq+9EY6n4UAj8kOj4kpqZbQTWSjojaBoALCLF3pcTjSMV35MYQ/nmoamUek+O8Y2xpOD7sgboL6mhJPH178lHwPVBn7i8J35W22mQ1IXozACih0VeMbPfSGoJvA50Ivrm/tjMtoVUZrWcZCx/JHrowIBVwF0Vx+STmaRewDggC1hB9IyjNFLvfalsHE+Smu9JQ2At0MXMdgZtKfe7AiccS8r9rkj6/4F/BsqAL4E7iX6nMwloEbTdbGYHa3S/HjzOOecSyQ+1OeecSygPHueccwnlweOccy6hPHicc84llAePc865hPLgcc45l1AePM6lCEkvSLq+6p7OJTcPHueccwnlwePcaZCUF9ygbWxwQ633JTWQNE1SJOiTE9zrCEk/kfRnSe9IWilppKRRwdWnCyW1qOZ++0j6OLiS+Hsx1zsbLmmWpLmS3gouh9JU0Zv8pQV9GkpaKylTUldJfw+286mkM4M+Pw5uDjZX0idx+Y/n6iwPHudOXwHwtJmdDewA/lsV/XsCNxK9iORvgH3B1ac/B26tameSMoHfA9ebWR9gfLAdiN7A6wIzOw9YDAwLLukyF7gs6PNPwHvBhSHHAD8LtvMvwB+CPv8OfDfYzqCqanLu28iouotzrgorzWxO8PNsIK+K/h+Z2W5gt6SdwDtB+3zg3Grs7wyi4TUlem1H0olehh+gp6T/DTQDGgPvBe2vEb0m10dE7yHzB0mNiV6N+I1gOwD1gud/AC8Ed6V8uxo1OVdtHjzOnb7YCyiWAw2IXnSx4ohC/ZP0PxLz+gjV+50UsNDMLqxk2QvAdWY2V9JPgMuD9snA/w0O5fUBPiR6/5UdZtbr2I2Y2d2S+gHfB+ZI6mVmW6tRm3NV8kNtzsXHKqIf8PD1JeZryhIgV9KFED30JunsYFk2sCE4HHdTxQpmtgeYCTwBvBvcN2YXsFLSj4PtSNJ5wc9dzWyGmf07sAXoWMNjcHWYB49z8fE74B5J04GcmtywmR0iGmb/IWkuMIfoITOA/wXMAKYAXx2z6mvAzcFzhZuAYcF2FhK9IyXAf0qaL2kB8AnR74icqxF+WwTnnHMJ5TMe55xzCeUnFziXZCQ9DXznmOYnzGxCGPU4V9P8UJtzzrmE8kNtzjnnEsqDxznnXEJ58DjnnEsoDx7nnHMJ5cHjnHMuof4fWNMQ490pvY8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "n_leafs = len(num_leaves_s)\n",
    "import matplotlib.pyplot\n",
    "x_axis = num_leaves_s\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_means, yerr=test_stds,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_means, yerr=train_stds,label = ' Train')\n",
    "plt.xlabel( 'num_leaves' )\n",
    "plt.ylabel( 'Log Loss' )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### min_child_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:34:47.210614Z",
     "start_time": "2020-06-27T11:30:39.974281Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  3.2min remaining:  1.6min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  4.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=0.7,\n",
       "        importance_type='split', learning_rate=0.1, max_bin=127,\n",
       "        max_depth=7, min_child_samples=20, min_child_weight=0.001,\n",
       "        min_split_gain=0.0, n_estimators=335, n_jobs=4, num_class=9,\n",
       "        num_leaves=60, objective='multiclass', random_state=None,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'min_child_samples': range(10, 50, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':60,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(10,50,10) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:35:59.526350Z",
     "start_time": "2020-06-27T11:35:59.522362Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.487489427488\n",
      "{'min_child_samples': 30}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:36:03.975118Z",
     "start_time": "2020-06-27T11:36:03.906265Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt8VOWdx/HPLwkBAuEe7oQEBDSCAYlcDCrVsqKtYC3gFS2orHZdW6r10m23rVrbpSpg67Zru2C17SqgbfHSUkGpykUNSLhFCCQBwv1+FULgt3/MSFMakiEJOTPJ9/165UXOmefM/J7XIfOd85w55zF3R0REJC7oAkREJDooEEREBFAgiIhImAJBREQABYKIiIQpEEREBFAgiIhImAJBREQABYKIiIQlBF3A2WjTpo2npaUFXYaISExZsmTJLndPqaxdTAVCWloaOTk5QZchIhJTzGxDJO00ZCQiIoACQUREwhQIIiICKBBERCRMgSAiIoACQUREwhQIIiIC1INAcHdmfLyJt1dvD7oUEZGoFlMXplVF6UnnxcVFbNl3lL5dLicluWHQJYmIRKU6f4TQID6OyWP6cuhYKY++thx3D7okEZGoVOcDAaBHu2QeHn4+c/N2MDOnOOhyRESiUr0IBIBxl6YxuFtrfvj6KjbtORJ0OSIiUSeiQDCz4Wa2xszWmdkjFbQbZWZuZlnh5UQzm25mK8ws18yGlmk7P/ycy8I/bavdmwrExRlPjckkzowHZuRy4qSGjkREyqo0EMwsHngOuAbIAG42s4xy2iUD9wMflll9N4C79wGGAU+bWdnXvNXd+4Z/dlS9G5Hp1KIx3x9xIR8V7eF/Pyg41y8nIhJTIjlCGACsc/cCdy8BXgZGltPucWAScLTMugxgHkD4DX8fkFWtiqvpqxd34uoL2/HUnLV8uu1AkKWIiESVSAKhE7CpzHJxeN0pZtYP6OLub5y2bS4w0swSzCwd6A90KfP49PBw0ffMzM6+/LNnZjz5lT40a5zAxFdyKSk9WRsvKyIS9SIJhPLeqE8NwIeHgCYDD5TTbhqhAMkBpgALgdLwY7eGh5IuC/+MLffFzSaYWY6Z5ezcuTOCcivXumlDfnzDReRtPcCUuWtr5DlFRGJdJIFQzD9+qu8MbCmznAz0BuabWREwCJhtZlnuXuruE8PnCEYCLYB8AHffHP73IPB7QkNT/8Tdn3f3LHfPSkmpdAa4iA3LaMeYrM788m/rWbJhT409r4hIrIokED4GephZupklAjcBsz9/0N33u3sbd09z9zRgMTDC3XPMLMnMmgCY2TCg1N1Xh4eQ2oTXNwC+DKys2a5V7ntfzqBji8Z8a0Yuh4+VVr6BiEgdVmkguHspcB8wB8gDZrj7KjN7zMxGVLJ5W2CpmeUBD/P3YaGGwBwzWw4sAzYDv6piH6osuVEDnh6dycY9R3jyrbzafnkRkahisXQrh6ysLM/Jyanx533yrTyef6+A6eMu4Qu9zunlECIitc7Mlrh7pd/wrDdXKlfkW8N60qtdMg/PWs7ewyVBlyMiEggFAtCoQTzP3JjJ3iMlfPdPK3UDPBGplxQIYRd2bM43v9iTN5dvZXbulso3EBGpYxQIZfzr5d24OLUF3/vjSrbu/yzockREapUCoYyE+DieGdOX4yech2Zp7gQRqV8UCKdJa9OE//jSBbyfv4uXFm8IuhwRkVqjQCjHrQNTuaJnCk++lUfBzkNBlyMiUisUCOUwMyaNuoiGCfFMnJFL6QndAE9E6j4Fwhm0a9aIJ67vTe6mffxi/vqgyxEROecUCBW4LrMjIzI7MnVePiuK9wddjojIOaVAqMTjI3vTumkiE2cs4+jxE0GXIyJyzigQKtE8qQE/HZXJuh2H+OmcNUGXIyJyzigQInB5zxRuH9yV//2gkIXrdwVdjojIOaFAiNAj15xPepsmfHvmcg4cPR50OSIiNU6BEKGkxASeGZPJ1v2f8djrq4MuR0SkxikQzkK/1Jb82xfOY9aSYuas2hZ0OSIiNUqBcJb+/coe9O7UjEdfW8HOg8eCLkdEpMZEFAhmNtzM1pjZOjN7pIJ2o8zMzSwrvJxoZtPNbIWZ5ZrZ0HK2mW1mtT6fclUlJsQxeUxfDh0r5dHXdAM8Eak7Kg0EM4sHngOuATKAm80so5x2ycD9wIdlVt8N4O59gGHA02YWV2abG4CYu1lQj3bJPHR1L+bm7WBmTnHQ5YiI1IhIjhAGAOvcvcDdS4CXgZHltHscmAQcLbMuA5gH4O47gH3A50cPTYFvAU9UufoAjc9OZ1C3Vvzw9VVs2nMk6HJERKotkkDoBGwqs1wcXneKmfUDurj7G6dtmwuMNLMEM0sH+gNdwo89DjwNVPhuamYTzCzHzHJ27twZQbm1Iy7OeGp0JmbGAzNzOXFSQ0ciEtsiCQQrZ92pd7/wENBk4IFy2k0jFCA5wBRgIVBqZn2B89z9D5W9uLs/7+5Z7p6VkpISQbm1p3PLJL5/XQYfFe5h2geFQZcjIlItkQRCMX//VA/QGSg76XAy0BuYb2ZFwCBgtplluXupu090977uPhJoAeQDg4H+4fYfAD3NbH51OxOEUf078y8Z7fjpnDWs2XYw6HJERKoskkD4GOhhZulmlgjcBMz+/EF33+/ubdw9zd3TgMXACHfPMbMkM2sCYGbDgFJ3X+3uv3D3juH2Q4C17j60ZrtWO8yMJ2/oQ7PGCUx8ZRklpZo7QURiU6WB4O6lwH3AHCAPmOHuq8zsMTMbUcnmbYGlZpYHPAyMrW7B0ahN04Y8+ZU+rN56gKnz1gZdjohIlVgsfY8+KyvLc3Jygi7jjL49M5dXlxYz855L6d+1ZdDliIgAYGZL3D2rsna6UrkG/ed1GXRo3pgHZizjSElp0OWIiJwVBUINSm7UgKfHZLJhzxGefCsv6HJERM6KAqGGDerWmruGpPPbxRuZv2ZH0OWIiERMgXAOPPAvvejZrikPzVrOviMlQZcjIhIRBcI50KhBPM+M6cuewyV8948xc98+EannFAjnSO9OzfnmF3vwxvKtzM7dUvkGIiIBUyCcQ/dc0Z1+qS347h9WsG3/0co3EBEJkALhHEqIj+OZMX05fsL59qxczZ0gIlFNgXCOpbdpwne+dAHv5+/it4s3BF2OiMgZKRBqwW0DU7m8Zwo/eiuPwl2Hgy5HRKRcCoRaYGb8dNRFNEyIZ+Iryyg9oRvgiUj0USDUknbNGvHE9b1Ztmkfv5i/PuhyRET+iQKhFl2X2ZHrMjsydV4+KzfvD7ocEZF/oECoZY+PvJDWTROZ+Moyjh4/EXQ5IiKnKBBqWYukRCaNyiR/xyGemrMm6HJERE5RIATgip4pjB3Ulf9dUMii9buDLkdEBIgwEMxsuJmtMbN1ZvZIBe1GmZmbWVZ4OdHMppvZCjPLNbOhZdr+JbxulZn90sziq92bGPLoteeT1roJD87M5eDR40GXIyJSeSCE36ifA64BMoCbzSyjnHbJwP3Ah2VW3w3g7n2AYcDTZvb5a45x90ygN5ACjK5GP2JOUmICT4/JZOv+z/jh66uDLkdEJKIjhAHAOncvcPcS4GVgZDntHgcmAWVv2pMBzANw9x3APiArvHwg3CYBSATq3X0dLk5tydeHnsesJcXMWbUt6HJEpJ6LJBA6AZvKLBeH151iZv2ALu7+xmnb5gIjzSzBzNKB/kCXMtvNAXYAB4FZZ19+7Lv/qh5c2LEZ33ltBbsOHQu6HBGpxyIJBCtn3alP8+EhoMnAA+W0m0YoQHKAKcBC4NRkw+5+NdABaAhcWe6Lm00wsxwzy9m5c2cE5caWxIQ4Jt/Yl4PHSnn0tRW6AZ6IBCaSQCimzKd6oDNQ9gb/yYTOA8w3syJgEDDbzLLcvdTdJ7p7X3cfCbQA8ss+ubsfBWZT/jAU7v68u2e5e1ZKSkqk/YopPdsl89DVvXh79XZmLikOuhwRqaciCYSPgR5mlm5micBNhN7AAXD3/e7ext3T3D0NWAyMcPccM0sysyYAZjYMKHX31WbW1Mw6hNcnANcCn9Zs12LL+Ox0Bqa34rHXV7Npz5GgyxGReqjSQHD3UuA+YA6QB8xw91Vm9piZjahk87bAUjPLAx4GxobXNyF0FLGc0HmGHcAvq9iHOiEuznhqdCYAD87M5eRJDR2JSO2yWBqzzsrK8pycnKDLOKdm5GzioVnL+e6XLuCuy7oFXY6I1AFmtsTdsyprpyuVo8zo/p0ZltGOSXPWsHb7waDLEZF6RIEQZcyMH9/Qh+SGCUx8ZRklpZo7QURqhwIhCrVp2pAf39CHVVsO8Oy8/Mo3EBGpAQqEKPUvF7ZnVP/O/Pf8dSzduDfockSkHlAgRLHvX5dBh+aN+dYryzhSUlr5BiIi1aBAiGLJjRrw1OhMNuw5wo/fqteXaYhILVAgRLnB3VtzZ3Y6Ly3ewN/W1r1bd4hI9FAgxIAHr+5Fj7ZN+fbMXPYdKQm6HBGpoxQIMaBRg3gm39iXPYdL+N6fVgVdjojUUQqEGNG7U3O++cUevJ67hdm5WyrfQETkLCkQYsg9V3SnX2oLvvfHlWzbf7TyDUREzoICIYYkxMfxzJi+lJSe5KFXl2vuBBGpUQqEGJPepgnfufZ83lu7k99+uDHockSkDlEgxKDbBnXl8p4pPPlmHoW7DgddjojUEQqEGGRmTPrqRSQmxPGtGcsoPaEb4IlI9SkQYlT75o14/PrefLJxH7/82/qgyxGROiCiQDCz4Wa2xszWmdkjFbQbZWZuZlnh5UQzm25mK8ws18yGhtcnmdmbZvapma0ys5/USG/qmRGZHfnyRR2YMjeflZv3B12OiMS4SgPBzOKB54BrgAzgZjPLKKddMnA/8GGZ1XcDuHsfYBjwtJl9/ppPufv5QD8g28yuqU5H6qsnru9NqyaJTHxlGUePnwi6HBGJYZEcIQwA1rl7gbuXAC8DI8tp9zgwCSj7BfkMYB6Au+8A9gFZ7n7E3d8Nry8BlgKdq9yLeqxFUiKTRl1E/o5DPP3XNUGXIyIxLJJA6ARsKrNcHF53ipn1A7q4+xunbZsLjDSzBDNLB/oDXU7btgVwHeHgkLM3tFdbbhuUyq8/KGRxwe6gyxGRGBVJIFg5605dERUeApoMPFBOu2mEAiQHmAIsBErLbJsA/B/wrLsXlPviZhPMLMfMcnbu1N0+z+Q7115A11ZJPDAjl4NHjwddjojEoEgCoZh//FTfGSh7M51koDcw38yKgEHAbDPLcvdSd5/o7n3dfSTQAig7J+TzQL67TznTi7v78+6e5e5ZKSkpkfWqHkpKTODpMX3Zuv8zHnt9ddDliEgMiiQQPgZ6mFm6mSUCNwGzP3/Q3fe7ext3T3P3NGAxMMLdc8LfJmoCYGbDgFJ3Xx1efgJoDnyzZrtUf/Xv2pJ7h3Zn5pJi/rpqW9DliEiMqTQQ3L0UuA+YA+QBM9x9lZk9ZmYjKtm8LbDUzPKAh4GxAGbWGfgPQiedl5rZMjO7qxr9kLBvXNWTjA7NePS1Few6dCzockQkhlgs3SAtKyvLc3Jygi4j6q3ZdpDrfvYBV/RK4fmx/TEr7zSQiNQXZrbE3bMqa6crleugXu2T+fbVvXh79XZmLSkOuhwRiREKhDpq/JB0BqS34oevr6Z475GgyxGRGKBAqKPi44ynR2fi7jw4M5eTJ2NnaFBEgqFAqMO6tEri+9ddyOKCPUxbUBh0OSIS5RQIddzorM588YJ2TJqzhrXbDwZdjohEMQVCHWdm/OSrfUhumMDEV5ZRUqq5E0SkfAqEeqBN04Y8eUMfVm05wM/eya98AxGplxQI9cTVF7ZnVP/OPPfuOpZu3Bt0OSIShRQI9ch/XpdBh+aNeWBGLkdKSivfQETqFQVCPdKsUQOeGp1J4a7D/PitT4MuR0SijAKhnhncvTV3DknnpcUb+Nta3U5cRP5OgVAPffvqXvRo25SHZuWy70hJ0OWISJRQINRDjRrEM/nGvuw+VMJ//mlV0OWISJRQINRTvTs15xtX9WB27hZez91S+QYiUucpEOqxe4d2p2+XFnz3jyvZfuBo0OWISMAUCPVYQnwcz4zJ5FjpCb49azmxNDeGiNQ8BUI91y2lKd+59gLeW7uT3324MehyRCRAEQWCmQ03szVmts7MHqmg3SgzczPLCi8nmtl0M1thZrlmNrRM2x+Z2SYzO1TtXki1jB3Ulct6tOFHb+ZRuOtw0OWISEAqDQQziweeA64hNAfyzWaWUU67ZOB+4MMyq+8GcPc+wDDgaTP7/DVfBwZUq3qpEWbGT0dl0iDeeGDGMkpP6AZ4IvVRJEcIA4B17l7g7iXAy8DIcto9DkwCyp6dzADmAbj7DmAfkBVeXuzuW6tRu9Sg9s0b8fj1vVm6cR//815B0OWISAAiCYROwKYyy8XhdaeYWT+gi7u/cdq2ucBIM0sws3SgP9DlbAo0swlmlmNmOTt36srac2lEZke+dFEHJr+9lpWb9wddjojUskgCwcpZd+rrKOEhoMnAA+W0m0YoQHKAKcBC4Kzuqubuz7t7lrtnpaSknM2mcpbMjCdG9qZVk0S+NWMZR4+fCLokEalFkQRCMf/4qb4zUPZKpmSgNzDfzIqAQcBsM8ty91J3n+jufd19JNAC0A35o1jLJon816iLWLv9EM+8vTbockSkFkUSCB8DPcws3cwSgZuA2Z8/6O773b2Nu6e5exqwGBjh7jlmlmRmTQDMbBhQ6u6ra74bUpO+0Ksttw5M5VfvF7C4YHfQ5YhILak0ENy9FLgPmAPkATPcfZWZPWZmIyrZvC2w1MzygIeBsZ8/YGaTzKwYSDKzYjP7QVU7ITXvO9deQGqrJB6cmcvBo8eDLkdEaoHF0tWpWVlZnpOTE3QZ9caSDXsY/ctFjOrfmUmjMoMuR0SqyMyWuHtWZe10pbKcUf+urbjniu7MyCnmr6u2BV2OiJxjCgSp0De/2JOMDs149LUV7Dp0LOhyROQcUiBIhRIT4ph8Y18OHi3lO6+t0A3wROowBYJUqlf7ZB68uid/Xb2dV5duDrocETlHFAgSkTuHdGNAeit+MHsVxXuPBF2OiJwDCgSJSHyc8fToTNydB2fmcvKkho5E6hoFgkSsS6skvn/dhSwu2MO0BYVBlyMiNUyBIGdldFZnvnhBWybNWUP+9oNBlyMiNUiBIGfFzPjxDRfRtGECE2cso6RUcyeI1BUKBDlrKckNefIrfVi5+QA/f0f3KhSpKxQIUiXDe7fnqxd35rn56/lk496gyxGRGqBAkCr7/ogM2jdrxLdm5PJZieZOEIl1CgSpsmaNGvDT0RdRuOswP/5zXtDliEg1KRCkWi7t3obx2em8uGgD763VFKcisUyBINX20PBenNe2KQ/NWs7+I5o7QSRWKRCk2ho1iGfymL7sOnSM/5y9MuhyRKSKIgoEMxtuZmvMbJ2ZPVJBu1Fm5maWFV5ONLPpZrbCzHLNbGiZtv3D69eZ2bNmZtXujQSmT+fm3H9VD/60bAtvLN9S+QYiEnUqDQQziweeA64BMoCbzSyjnHbJwP3Ah2VW3w3g7n2AYcDTZvb5a/4CmAD0CP8Mr3o3JBp8fWh3Mru04Lt/XMn2A0eDLkdEzlIkRwgDgHXuXuDuJcDLwMhy2j0OTALKvhNkAPMA3H0HsA/IMrMOQDN3X+ShG+y/CFxf9W5INEiIj+OZMZkcPX6Ch19drrkTRGJMJIHQCdhUZrk4vO4UM+sHdHH3N07bNhcYaWYJZpYO9Ae6hLcvrug5yzz3BDPLMbOcnTv1LZZo1z2lKY9ecwHz1+zk9x9tDLocETkLkQRCeWP7pz76hYeAJgMPlNNuGqE3+xxgCrAQKK3sOf9hpfvz7p7l7lkpKSkRlCtBGzuoK5f1aMMTb+RRtOtw0OWISIQiCYRiQp/qP9cZKHvWMBnoDcw3syJgEDDbzLLcvdTdJ7p7X3cfCbQA8sPP2bmC55QYFhdnTBp1EQ3ijW/NWEbpCd0ATyQWRBIIHwM9zCzdzBKBm4DZnz/o7vvdvY27p7l7GrAYGOHuOWaWZGZNAMxsGFDq7qvdfStw0MwGhb9ddDvwpxrumwSoQ/PGPH59b5Zu3Mf/vFcQdDkiEoFKA8HdS4H7gDlAHjDD3VeZ2WNmNqKSzdsCS80sD3gYGFvmsXuBXwPrgPXAn6tQv0SxEZkd+VKfDkyZu5ZVW/YHXY6IVMJi6ZsgWVlZnpOTE3QZchb2Hi7hX6a8R6ukRP50XzaNGsQHXZJIvWNmS9w9q7J2ulJZzqmWTRKZNOoi1mw/yDNvrw26HBGpgAJBzrkv9GrLLQNT+dX7BXxYsDvockTkDBQIUiv+49oLSG2VxAMzczl4VDfAE4lGCgSpFU0aJvDMmEy27PuMJ97Q3Aki0UiBILWmf9dW3HNFd17J2cTbq7cHXY6InEaBILXqm1/syQUdmvHoa8vZfehY0OWISBkKBKlViQlxTL4xkwOflfLIaysoKdVVzCLRQoEgte789s14aHgv3l69nS88NZ/fLt7AsdITQZclUu8pECQQdw5J5zfjB9C2WUO++8eVfOGn83lJwSASKF2pLIFyd97P38WUuWtZunEfHZo34utDuzPmki40TNBVzSI1IdIrlRUIEhXcnQ/W7WLK3HyWbNirYBCpQQoEiUnuzoJ1u5kydy05G/bSvlkjvv6F7ozJ6qL7IIlUkQJBYpq7s3B9KBg+LtpLu2YN+frQ87jxEgWD1D87Dh6lbXKjKm+vQJA6wd1ZtH43U+bm81HRHto1a8i9V3TnpgGpCgap006cdN5evY1pHxSxeusBFj16JcmNGlTpuSINhIQqPbtILTEzLj2vDYO7t2ZRQSgYfvD6av57/nruHdqdmxUMUsfs/+w4Mz7exAsLi9i87zM6t2zMN67qQZyVN/NwzdIRgsScReGhpA8L95CSHDpiuGWggkFiW8HOQ7ywsIhZS4o5UnKCAemtGJ+dzrCMdsTHVS8ManTIyMyGA1OBeODX7v6TM7QbBcwELglPodmA0KxoFxM6GnnR3X8cbvsN4G7AgF+5+5TK6lAgSFmL1u9m6ry1LC4IBcM9V3TnVgWDxJDPv3Y9fUEh767ZSWJ8HNdldmRcdhq9OzWvsdepsUAws3hgLTAMKCY0x/LN7r76tHbJwJtAInBfOBBuITS/8k1mlgSsBoYCTYGXgQFACfAX4F53z6+oFgWClGdxwW6mzs1nUcFu2jRtyD1XdOPWgV1pnKhgkOj0WckJXvukmBcWFJG/4xBtmjbktkGp3DqwKynJDWv89WryHMIAYJ27F4Sf+GVgJKE397IeByYBD5ZZ50ATM0sAGhN68z8AXAIsdvcj4ef8G/CV8PYiZ2VQt9YMmtCaDwt2M3VePk+8mccv/1agYJCos2XfZ7y4aAMvf7yRfUeOc2HHZjw9OpMvZ3aIiuttIgmETsCmMsvFwMCyDcysH9DF3d8ws7KBMItQeGwFkoCJ7r7HzFYCPzKz1sBnwLWAPvpLtQzs1prfd2vNR4V7mDpvbTgY1vOvl3fn1kGpJCXqOxRS+9ydpRv3MW1BIX9ZuQ135+oL2zMuO51L0lpitXCyOFKR/IWUV+2pcSYziwMmA18rp90A4ATQEWgJvG9mc909z8z+C3gbOATkAqXlvrjZBGACQGpqagTlSn03IL0Vv7trEB8X7WHq3Hx+9FYe//PeeiZc3o3bBnVVMEitKCk9yZ9XbmXagiJyN+0juVEC47PTuH1wGl1aJQVdXrkiOYcwGPiBu18dXn4UoMzJ4ebAekJv7ADtgT3ACGAcoaGhl8JtpwF/cfcZp73Gk0Cxu/93RbXoHIJURU7RHqbOy+f9/F20bpLIhMu7MXawgkHOjd2HjvF/H23kxUUb2HHwGN3aNGFcdho3XNyZJg2D+T9XkyeVEwidVL4K2EzopPIt7r7qDO3nAw+GTyo/DJwPjCc0ZPQxcJO7Lzeztu6+w8xSgb8Cg919b0W1KBCkOpZs2MOUuX8Phrsv78bYQV0D+yOVuuXTbQeY/kERf1i2mZLSk1zWow3jh6RzRY8U4qr5tdHqqrGTyu5eamb3AXMIfe10mruvMrPHgBx3n13B5s8B04GVhIaeprv78vBjr4bPIRwH/q2yMBCprv5dW/HSnQNZsmEvU+fl85M/f8rz7xVw92XduH2wgkHO3smTzjuf7mDagkIWrt9NowZxjOrfmXGXptGjXXLQ5Z01XZgm9dbSjXuZOjefv63dScukBtx9eTduH5xGUwWDVOLg0ePMWlLMCwuL2LD7CB2aN+L2wWncPKALLZISgy7vn+heRiIR+mRj6Ihh/hoFg1Rsw+7DvLCwiJk5xRw6Vkr/ri0Zl53G1Re2p0F89M43pkAQOUvLNu1j6ty1vLtmJy2SGnD3Zd2441IFQ33n7iwq2M30BUXMzdtOvBlfuqgD47LT6dulRdDlRUSBIFJFyzbt49l5+bzz6Y5TwXD74K5VvtOkxKajx08we9kWpi0o5NNtB2nVJJFbBqQydnBX2jWr+q2og6BAEKmm3E37mBoOhuaNG3D3ZenccWmagqGO237gKL9dvIHff7iR3YdLOL99MuOy0xjZt1PM3idLgSBSQ5YX72Pq3HzmhYPhriHp3JGdRjMFQ52yvHgf0z4o5M0VWyk96Vx1fjvGZ6cxuHvrqLqauCoUCCI1bEXxfqbOW8vcvB00a5TAXZd142sKhphWeuIkc1ZtZ9qCQpZs2EvThgmMzurMHYPTSGvTJOjyaowCQeQcCQVDPnPzttOsUQJ3DgkFQ/PGCoZYse9ICS9/vIkXFxaxZf9RUlsl8bVL0xid1blODgkqEETOsZWbQ8Hw9urtJDdK4M4h6YzLTlcwRLF1Ow4yfUERry4t5ujxkwzu1prxQ9K58vy21Z6EJpopEERqycrN+3l2Xj5/DQfD+Ox0xg9RMESLkyedv+XvZPqCIt5bu5PEhDiu79uRcdnpXNChWdDl1QoFgkgtW7UlFAxzVoWCYVx2Ondmp9M8ScEQhMPHSnltaTHTFxZRsPMwbZMbMnZQV24ZmErrpjU/CU00UyCIBGT1lgM8Oy+fv6zaRnLDBMZlp3HnkG4KhlpSvPfgtkRjAAAK+ElEQVRIaBKajzZy4GgpF3VuzvjsdK7t04HEhOi9mvhcUiCIBCxvaygY/rwyFAxfy07jziHpUXmvm1jn7uRs2Mu0DwqZs2obZsbw3u0Zn53GxanRNQlNEBQIIlEib+sBfvZOPm+t2EbThgl87dI07rpMwVATjpWe4M3lW5m2oJCVmw/QvHEDbh6Qyu2Du9KxReOgy4saCgSRKPPptgP8bN463lyxlaYNE7jj0q7cNaQbLZsoGM7WrkPH+N3ijby0eAO7Dh3jvLZNGZedxlf6ddLER+VQIIhEqTXbDvLsO/m8tWIrSQ3iuePSNO66rButFAyVWrVlP9MXFDF72RZKTpxkaK8Uxmenc1mPNvV+WKgiCgSRKLd2+0GenZfPmwqGCp046czN2860Dwr5sHAPjRvEM6p/Z76WnUb3lKZBlxcTajQQzGw4MJXQjGm/dvefnKHdKGAmcEl4Cs0GwK+BiwnNzvZimbmYJwJ3AQ6sAMa5+9GK6lAgSF2Uv/0gz76zjjeWb6FxOBjuVjBw4OhxZny8id8sKmLTns/o1KIxd1zalRuzUvWNrbNUk3MqxxOaU3kYUExoXuSb3X31ae2SgTeBROC+cCDcAoxw95vMLAlYDQwlNG3mB0CGu39mZjOAt9z9hYpqUSBIXZa//SA/e2cdr4eD4fbBadx9WXq9+8584a7DvLCgkFlLijlccoIBaa0Yl53GsIx2JETxJDTRrMbmVAYGAOvcvSD8xC8DIwm9uZf1ODAJeLDMOgeamFkC0BgoAQ6Ef08AGpvZcSAJ2BJBLSJ1Vo92yTx7cz/uv+o8fvbOOv7nvfW8uKiIsYO7MuGybnU6GNydBet2M31BIe+s2UFCnHFdZkfGZ6fTu1PzoMurNyIJhE7ApjLLxcDAsg3MrB/Qxd3fMLOygTCLUHhsJfSmP9Hd94S3eQrYCHwG/NXd/1rlXojUIee1TWbqTf349yt78PN38vnVewW8uHADtw/uyt2Xd6NNHQqGo8dP8IdPNjN9QSFrtx+iTdNE7r+yB7cOSqVtcmxNQlMXRBII5Z26PzXOZGZxwGTga+W0GwCcADoCLYH3zWwusJdQUKQD+4CZZnabu//2n17cbAIwASA1NTWCckXqhvPaNmXKTf3496t68PN31vGr9wt4cdGG0BFDjAfD1v2f8dKiDfzfRxvZe+Q4GR2a8dNRF3FdZseYnYSmLojkHMJg4AfufnV4+VGAMieHmwPrgUPhTdoDe4ARwDhgsbu/FG47DfgLoUAZ7u53htffDgxy969XVIvOIUh9VrDzED9/Zx1/XLaZxIQ4xg7qyoTLu5OSHDvBsHTjXqYvKOLPK7Zy0p1hGe0Yn53OgPRW+troOVSTJ5UTCJ1UvgrYTOik8i3uvuoM7ecDD4ZPKj8MnA+MJzRk9DFwE6FzCNOASwgNGb0A5Lj7zyqqRYEg8s/BcNvArky4olvUDrEcP3GSP6/cxrQPClm2aR/JDRO48ZIu3HFpGl1aJQVdXr1QYyeV3b3UzO4D5hD62uk0d19lZo8RehOfXcHmzwHTgZWEhp6mu/vycIGzgKVAKfAJ8HxltYgIdEtpyjM39uXfr+rBz97JZ9qCQl5avIHbBnXlX6MoGPYeLuH3H23kpUUb2HbgKGmtk/jhiAv5av/ONG2oq4mjkS5ME4lxhbsOnzpiSIgzbh3YlXuu6EbbZsEEw5ptB3lhYSGvLd3MsdKTDDmvDeOHpDG0Z1vi6vAkNNFMVyqL1DNFuw7z83fX8YdPQsFwy8BU7r2ie60Ew8mTzrtrdjB9QREfrNtFw4Q4bri4M+Oy0+jZLvmcv75UTIEgUk9t2B06YngtHAw3D0jl3qHdaXcOguHQsVJm5WziN4s2ULjrMO2bNeL2S7ty8yWpumlfFFEgiNRzG3cf4efv5vPq0s3Exxm3DEjlniu607559YNh054jvLCwiBkfb+LgsVL6pbZgXHY61/RuTwNdTRx1FAgiAoSC4bl31/Hq0mLi4oybL+nCvUPPO+tgcHc+LNzDtA8KmZu3nTgzru3TgXHZafRLbXmOqpeaoEAQkX+waU8oGGYtKSbOjJsGdOHeod3p0LziiWSOHj/B67lbmLagiLytB2iZ1IBbBqYydlBajRxtyLmnQBCRcm3ac4T/nr+OmTkVB8OOg0f57eKN/G7xBnYfLqFnu6aMz07n+n6ddDVxjFEgiEiFQsGwnpk5m4gz48ZLQsGw+1AJ0xcU8vryLZSedK7s1ZbxQ9K5tHtrXU0coxQIIhKR4r1/D4aTHpqQpkliPKOzQlcTp7dpEnSJUk0KBBE5K5v3hW4416ZpImMu6UKzRpqEpq6oyfkQRKQe6NSiMY9cc37QZUiA9IVhEREBFAgiIhKmQBAREUCBICIiYQoEEREBFAgiIhKmQBAREUCBICIiYTF1pbKZ7QQ2VHHzNsCuGiwnSHWlL3WlH6C+RKu60pfq9qOru6dU1iimAqE6zCwnkku3Y0Fd6Utd6QeoL9GqrvSltvqhISMREQEUCCIiElafAuH5oAuoQXWlL3WlH6C+RKu60pda6Ue9OYcgIiIVq09HCCIiUoE6GQhmNs3MdpjZyjLrWpnZ22aWH/63ZZA1RuIM/fiBmW02s2Xhn2uDrDFSZtbFzN41szwzW2Vm3wivj6n9UkE/Ym6/mFkjM/vIzHLDfflheH26mX0Y3ievmFli0LVWpoK+vGBmhWX2S9+ga42EmcWb2Sdm9kZ4uVb2SZ0MBOAFYPhp6x4B5rl7D2BeeDnavcA/9wNgsrv3Df+8Vcs1VVUp8IC7XwAMAv7NzDKIvf1ypn5A7O2XY8CV7p4J9AWGm9kg4L8I9aUHsBe4M8AaI3WmvgB8u8x+WRZciWflG0BemeVa2Sd1MhDc/T1gz2mrRwK/Cf/+G+D6Wi2qCs7Qj5jk7lvdfWn494OE/rN3Isb2SwX9iDkecii82CD848CVwKzw+qjfJ1BhX2KOmXUGvgT8Orxs1NI+qZOBcAbt3H0rhP6ogbYB11Md95nZ8vCQUlQPsZTHzNKAfsCHxPB+Oa0fEIP7JTw0sQzYAbwNrAf2uXtpuEkxMRJ4p/fF3T/fLz8K75fJZtYwwBIjNQV4CDgZXm5NLe2T+hQIdcUvgO6EDou3Ak8HW87ZMbOmwKvAN939QND1VFU5/YjJ/eLuJ9y9L9AZGABcUF6z2q2qak7vi5n1Bh4FzgcuAVoBDwdYYqXM7MvADndfUnZ1OU3PyT6pT4Gw3cw6AIT/3RFwPVXi7tvD//FPAr8i9EccE8ysAaE30d+5+2vh1TG3X8rrRyzvFwB33wfMJ3RepIWZJYQf6gxsCaquqijTl+HhIT5392PAdKJ/v2QDI8ysCHiZ0FDRFGppn9SnQJgN3BH+/Q7gTwHWUmWfv3mGfQVYeaa20SQ8Dvq/QJ67P1PmoZjaL2fqRyzuFzNLMbMW4d8bA18kdE7kXWBUuFnU7xM4Y18+LfNhwwiNu0f1fnH3R929s7unATcB77j7rdTSPqmTF6aZ2f8BQwndIXA78H3gj8AMIBXYCIx296g+YXuGfgwlNCzhQBHwr5+PwUczMxsCvA+s4O9jo98hNP4eM/ulgn7cTIztFzO7iNAJynhCHw5nuPtjZtaN0KfTVsAnwG3hT9hRq4K+vAOkEBp2WQbcU+bkc1Qzs6HAg+7+5draJ3UyEERE5OzVpyEjERGpgAJBREQABYKIiIQpEEREBFAgiIhImAJBREQABYKIiIQpEEREBID/B833jEmMInTAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 列采样参数 sub_feature/feature_fraction/colsample_bytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:49:02.294050Z",
     "start_time": "2020-06-27T11:43:47.234870Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  4.2min remaining:  1.1min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed:  5.2min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=1.0,\n",
       "        importance_type='split', learning_rate=0.1, max_bin=127,\n",
       "        max_depth=7, min_child_samples=30, min_child_weight=0.001,\n",
       "        min_split_gain=0.0, n_estimators=335, n_jobs=4, num_class=9,\n",
       "        num_leaves=60, objective='multiclass', random_state=None,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':60,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:49:24.724992Z",
     "start_time": "2020-06-27T11:49:24.721002Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48682580651\n",
      "{'colsample_bytree': 0.5}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:50:02.597219Z",
     "start_time": "2020-06-27T11:50:02.519362Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VPW9//HXJzuEsIY9QAIisolARBBCbdXLYgUtWper1wWh9Vevtcv9XfvrvW2vfdzN29al19teQJHWuhVaRUuhaqUEBCVEIwKikAQSwhK2sIZsn98fGeiIASaQ5EyS9/Px4MHMme+ZeZ9DmHfmzJnvmLsjIiISE3QAERGJDioEEREBVAgiIhKiQhAREUCFICIiISoEEREBVAgiIhKiQhAREUCFICIiIXFBB6iP1NRUT09PDzqGiEizkZqayrJly5a5++RzjW1WhZCenk5OTk7QMUREmhUzS41knA4ZiYgIoEIQEZEQFYKIiAAqBBERCVEhiIgIoEIQEZEQFYKIiAAqBBFpYoV7j/LK+zuortHX90YbFYKINKl/XPQhD730AbfPXUPxgWNBx5EwKgQRaTJ5RQd5t2A/1w7pzkc7ypjyeDa/yy3GXa8WooEKQUSazNzsfFKS4njslstY+tBEBvVI4dsv5/HA8+9z8FhF0PFaPRWCiDSJov3HWLJ+J7df0Zd2iXH06dyWl742jn+YNIhlG3Yx6fEVrPx0b9AxWzUVgog0iWdWFRBjxj1XZpxaFhtjfOOLF/HKN8aTkhTPHU+/y7+8toHyyuoAk7ZeKgQRaXRlxyp5aW0R0y7rRY8OSZ+7fVjvDrz+9xO4+8p05q8q5Pqfr+SjHWUBJG3dVAgi0uh+8942jlVUMyur/xnHJMXH8qNpQ1lw7xjKjldy4/+s4n+Wb9HpqU1IhSAijepEVTXPrioka2Aqg3u2P+f4L1zclWUPTeTaId15dOlmbpuzhqL9Oj21KURUCGY22cw2m9kWM3v4LONuMjM3s8zQ9QQzm29m680sz8yuChv7r2ZWZGZHLngrRCRqLf6ghD2HT5z11cHpOiUn8NTto/jpzSPYuPMQU57IZuE6nZ7a2M5ZCGYWCzwFTAGGALeZ2ZA6xqUADwLvhi2eBeDuw4FrgZ+a2cnHfA0Yc0HpRSSquTvzsgu4pEcKWQMj+tKuU8yMGaPT+OM3sxjSsz3f/W0e/+c3uRw4qtNTG0skrxDGAFvcPd/dK4AXgel1jPsx8ChQHrZsCPAWgLvvAQ4CmaHra9x95wVkF5Eot+LTvWzefZhZWf0xs/O6jz6d2/LC7LE8POUS3ty0m0mPr+Avn5Q2cFKByAqhN1AUdr04tOwUMxsJ9HH3109bNw+YbmZxZpYBjAb6XEBeEWlG5q7Ip3v7RK4f0euC7ic2xvj6FwbwyjfG07FtPHc98x4/fPUjjlfo9NSGFEkh1FXrpw7khQ4BPQZ8p45xz1BbIDnA48A7QFV9AprZbDPLMbOc0lL9ViDSXGwoKWPllr3cMz6DhLiGOX9laK8OLH5gAveOz2DB6m18+efZrC/W6akNJZJ/pWI++1t9GlASdj0FGAYsN7NCYCyw2Mwy3b3K3b/l7pe5+3SgI/BpfQK6+xx3z3T3zK5du9ZnVREJ0LzsApITYrltTN8Gvd+k+Fh+cP0Qnpt5BUdPVHPj/6ziv//8qU5PbQCRFMJaYKCZZZhZAnArsPjkje5e5u6p7p7u7unAGmCau+eYWVszSwYws2uBKnff2PCbISLRpOTgcV7LK+HWMX3p0Ca+UR5jwsBUlj6UxeRhPfjJnz7hq/+7mu37dHrqhThnIbh7FfAAsAzYBLzs7hvM7BEzm3aO1bsBuWa2CfhH4M6TN5jZo2ZWDLQ1s2Iz+9H5boSIRJdn3ynEgXvGpzfq43Rsm8DPbxvJ47dcxie7DzPliRW8nFOk01PPkzWnHZeZmek5OTlBxxCRszhUXsmV//5nvnRJN568bWSTPe6Og8f5zssfsCZ/P5OGduffbhxOl3aJTfb40czM1rl75rnG6ZPKItKgXnqviCMnqur1QbSG0LtjG56/byz/b+olvP1xKZMez+btj/c0aYbmToUgIg2msrqGZ1YVMLZ/Z4andWjyx4+JMWZPHMCrD4ynS3IC9zy7ln96Zb1OT42QCkFEGswfPtzJzrJyZk9s2lcHpxvcsz2vPjCe+yZk8Nya7Vz3ZDZ5RQcDzdQcqBBEpEG4O3NW5HNRt3ZcdXG3oOOQFB/LP315CM/fdwXlldXM+MU7PPnWp1RV1wQdLWqpEESkQazeuo+NOw8xKyuDmJjzm6aiMVx5USp/fGgi113ak5+98Qk3/+9qtu07GnSsqKRCEJEGMSc7n9R2iUy/rPe5BzexDm3ieeLWkTx520i27jnClCeyefG97To99TQqBBG5YJt3HWb55lLuvrIfSfGxQcc5o2kjerH0oYlc1qcjD/9uPbN+tY69R04EHStqqBBE5ILNy86nTXwsf3tFv6CjnFOvjm14buYV/NN1g1nxaSmTH1/BW5t2Bx0rKqgQROSC7DlUzisf7OCrmWl0Sk4IOk5EYmKM+7L689oDE0htl8jMBTl873frOVZRr7k3WxwVgohckGffKaS6xrl3QkbQUeptUI8UXn1gPF+b2J8X125n6hPZvL/9QNCxAqNCEJHzdvREFc+t2cbkYT3o1yU56DjnJTEulu9NHcwLs8ZSWe3c9MvVPPbGJ63y9FQVgoict5dzijhU3vTTVDSGsf278MeHspg+ohdPvPUpM365moK9rev0VBWCiJyXquoanl5ZwOXpnRjZt1PQcRpE+6R4fnbLZTx1+ygK9x5l6hPZ/Obdba3m9FQVgoicl6UbdlF84Dj3tYBXB6e77tKeLHtoIpnpnfj+7z/ivgU5lB5u+aenqhBEpN7cnbkr8slITeaawd2DjtMoenRIYsE9Y/jh9UPI3rKXyY+v4I2NLfv0VBWCiNTbewX7ySsuY+aEDGKjaJqKhhYTY9wzPoM//P0EurdPYtavcnh40YccPdEyT09VIYhIvc3NzqdzcgIzRqUFHaVJDOyewivfGM/9Vw3gpZwipj6ZzbptLe/0VBWCiNTL1tIjvLlpD3eO7UebhOidpqKhJcTF8I+TL+Gl2eOoqnZu/uU7/OxPm6lsQaenqhBEpF7mZReQGBfDneOif5qKxjAmozNLH8rixpFpPPnnLdz0i3fYWnok6FgNQoUgIhHbe+QEi3KLmTE6jdRW/H3FKUnx/PSrI/jF345i2/5jXPdkNr9e0/xPT1UhiEjEfrV6G5XVNcxshtNUNIYpw2tPTx2T0YV/fuUj7nl2LXsOlwcd67ypEEQkIscrqvn16kKuGdydAV3bBR0nanRvn8SCey7nX6YNZfXWfUx6bAVLP9oVdKzzokIQkYgszC3mwLHKwL8vORqZGXddmc4fHswirVNbvv7cOv7ht3kcaWanp6oQROScqmucp7PzuaxPRzL7tYxpKhrDRd3asej+K3ngixexKLeYKU+sIKdwf9CxIqZCEJFzemPjbgr3HWP2xP6YtdwPojWEhLgYvjtpEC9/bRwAX/3f1fzXso+pqIr+01NVCCJyTnOz8+nTuQ2ThvYIOkqzkZnemT9+cyI3jU7jqbe38pVfrGLLnsNBxzorFYKInNW6bftZt+0AM8e37GkqGkO7xDgevWkEv7xjNCUHy7nuyZUseKcwak9PVSGIyFnNXVFAhzbx3JzZJ+gozdbkYT1Y+lAW4wZ04YeLN3DX/LXsPhR9p6eqEETkjAr3HmXZxl3cMbYvyYlxQcdp1rqlJDH/7sv58Q3DeK9gH5MeX8Ef1+8MOtZnqBBE5IyeWVVAfEwMd41LDzpKi2Bm3Dm2H394MIt+ndty/29y+c7LeRwurww6GqBCEJEzOHC0gpdzirhhZC+6tU8KOk6LMqBrOxbefyUPXj2Q379fzJQnsnmvIPjTU1UIIlKn59Zso7yypkV+I1o0iI+N4dvXXszC+68kNsa4Zc5q/nNpsKenqhBE5HPKK6tZsLqQLw7qysXdU4KO06KN6tuJJQ9mcUtmH36xfCs3PLWKT3YHc3qqCkFEPueV93ew90gFszRNRZNITozjP2Zcyty/y2T3oXK+/POVPLOygJqapj09VYUgIp9RU+PMzc5nWO/2jOvfJeg4rcq1Q7qz9KGJZF2UyiOvb+Su+e+xq6zpTk9VIYjIZ7y9eQ9bS48yK0vTVASha0oi8+7K5N9uHE5O4QEmPb6C1z8saZLHjqgQzGyymW02sy1m9vBZxt1kZm5mmaHrCWY238zWm1memV0VNnZ0aPkWM3vS9JMnEhXmrMinV4ckpg7vGXSUVsvMuP2Kviz5ZhYZqcl8+6U8dhw83uiPe85CMLNY4ClgCjAEuM3MhtQxLgV4EHg3bPEsAHcfDlwL/NTMTj7mL4DZwMDQn8nnvxki0hDyig7ybsF+7p2QQXysDiAELSM1mYVfH8cLs8fSu2ObRn+8SP7FxwBb3D3f3SuAF4HpdYz7MfAoEH7AawjwFoC77wEOAplm1hNo7+6rvXZSj18BN5z/ZohIQ5ibnU9KYhy3XK5pKqJFXGwMo5toyvFICqE3UBR2vTi07BQzGwn0cffXT1s3D5huZnFmlgGMBvqE1i8+232G3fdsM8sxs5zS0tII4orI+Sjaf4wl63dy+xV9SUmKDzqOBCCSyUnqOrZ/6lyo0CGgx4C76xj3DDAYyAG2Ae8AVee6z88sdJ8DzAHIzMyMzikCRVqAZ1YVEGPG3ePTg44iAYmkEIqp/a3+pDQg/C3vFGAYsDz0vnAPYLGZTXP3HOBbJwea2TvAp8CB0P2c6T5FpAmVHavkpbVFTBvRi54dGv9YtUSnSA4ZrQUGmlmGmSUAtwKLT97o7mXunuru6e6eDqwBprl7jpm1NbNkADO7Fqhy943uvhM4bGZjQ2cX/R3wagNvm4hE6Pn3tnOsolrTVLRy53yF4O5VZvYAsAyIBZ5x9w1m9giQ4+6Lz7J6N2CZmdUAO4A7w267H3gWaAP8MfRHRJpYRVUN81cVkDUwlSG92gcdRwIU0QTn7r4EWHLash+cYexVYZcLgUFnGJdD7aEmEQnQ4rwS9hw+wU9uHhF0FAmYTjQWacXcnbkr8rmkRwpZA1ODjiMBUyGItGIrPt3L5t2HNU2FACoEkVZt7op8urdP5PoRvYKOIlFAhSDSSm0oKWPllr3cfWUGCXF6KhAVgkirNS+7gOSEWG6/om/QUSRKqBBEWqGSg8d5La+EWy7vS4c2mqZCaqkQRFqhZ98pxIF7NE2FhFEhiLQyh8sreeHd7Uwd3pM+ndsGHUeiiApBpJV5aW0Rh09UMSsrI+goEmVUCCKtSGV1Dc+sLGBs/85cmtYx6DgSZVQIIq3IkvU7KSkrZ/ZETWInn6dCEGkl3J05K/K5qFs7rrq4W9BxJAqpEERaidVb97Gh5BCzsjKIidE0FfJ5KgSRVmJOdj6p7RKZflmd31YrokIQaQ027zrM8s2l3DWuH0nxsUHHkSilQhBpBeZl55MUH8MdY/sFHUWimApBpIXbc6icVz7YwVcz+9ApOSHoOBLFVAgiLdyz7xRSVePMnKAPosnZqRBEWrCjJ6p4bs02Jg/tQb8uyUHHkSinQhBpwV7OKeJQeRWz9EE0iYAKQaSFqqqu4emVBWT268Sovp2CjiPNgApBpIVatmE3xQeO69WBREyFINIC1U5TsZWM1GSuGdw96DjSTKgQRFqgtYUHyCsuY+aEDGI1TYVESIUg0gLNWZFP5+QEZoxKCzqKNCMqBJEWZmvpEd7ctJs7x/ajTYKmqZDIqRBEWph52QUkxMVw5zhNUyH1o0IQaUH2HjnBotxiZoxKI7VdYtBxpJlRIYi0IL9avY2Kqhru0/cly3lQIYi0EMcrqvn16kKuGdydAV3bBR1HmiEVgkgLsTC3mAPHKvV9yXLeVAgiLUB1jfN0dj4j+nTk8nRNUyHnR4Ug0gK8sXE3hfuOMTurP2b6IJqcHxWCSAswLzufPp3bMGmopqmQ86dCEGnm1m07QM62A8wcn0FcrP5Ly/mL6KfHzCab2WYz22JmD59l3E1m5maWGboeb2YLzGy9mW0ys++Fjf2mmX1kZhvM7KEL3xSR1mledj4d2sRzc2afoKNIM3fOQjCzWOApYAowBLjNzIbUMS4FeBB4N2zxzUCiuw8HRgNfM7N0MxsGzALGACOAL5vZwAvdGJHWZtu+oyzdsIs7xvYlOTEu6DjSzEXyCmEMsMXd8929AngRmF7HuB8DjwLlYcscSDazOKANUAEcAgYDa9z9mLtXAX8Bbjz/zRBpnZ5eWUB8TAx3jUsPOoq0AJEUQm+gKOx6cWjZKWY2Eujj7q+ftu5C4CiwE9gO/MTd9wMfARPNrIuZtQWmAnq9K1IPB45W8HJOEdMv60W39klBx5EWIJLXmHWdw+anbjSLAR4D7q5j3BigGugFdAKyzexNd99kZv8JvAEcAfKAqjof3Gw2MBugb9++EcQVaR2eW7ON8soafSOaNJhIXiEU89nf3tOAkrDrKcAwYLmZFQJjgcWhN5ZvB5a6e6W77wFWAZkA7v60u49y94nAfuDTuh7c3ee4e6a7Z3bt2rV+WyfSQpVXVrNgdSFXDerKxd1Tgo4jLUQkhbAWGGhmGWaWANwKLD55o7uXuXuqu6e7ezqwBpjm7jnUHib6ktVKprYsPgYws26hv/sCXwFeaMDtEmnRXnl/B3uPVDA7S68OpOGc85CRu1eZ2QPAMiAWeMbdN5jZI0COuy8+y+pPAfOpfc/AgPnu/mHotkVm1gWoBL7h7gcuZENEWouaGmdudj5De7Vn3IAuQceRFiSi89TcfQmw5LRlPzjD2KvCLh+h9tTTusZlRZxSRE55e/MetpYe5YlbL9M0FdKg9LFGkWZmzop8enVIYurwnkFHkRZGhSDSjHxYfJB3C/Zz74QM4jVNhTQw/USJNCNzswtISYzjlsv1sR1peCoEkWaiaP8xlqzfye1X9CUlKT7oONICqRBEmon5qwox4O7x6UFHkRZKhSDSDJQdq+TFtdu5fkQvenZoE3QcaaFUCCLNwPPvbedYRTX3ZWUEHUVaMBWCSJSrqKph/qoCJlyUytBeHYKOIy2YCkEkyi3OK2HP4ROaxE4anQpBJIq5O3NX5DOoewoTB6YGHUdaOBWCSBRb8eleNu8+zKyJ/TVNhTQ6FYJIFJu7Ip/u7ROZNqJX0FGkFVAhiESpDSVlrNyyl7uvzCAhTv9VpfHpp0wkSj2dXUByQiy3X6FvCpSmoUIQiUI7y46zOK+EWy7vS4c2mqZCmoYKQSQKPbuqEAfu0TQV0oRUCCJR5nB5Jc+/u52pw3vSp3PboONIK6JCEIkyL60t4vCJKmZpmgppYioEkShSWV3DMysLuCKjM5emdQw6jrQyKgSRKLJk/U5KysqZrWkqJAAqBJEo4e7MWZHPgK7JfHFQt6DjSCukQhCJEqu37mNDySFmZfUnJkbTVEjTUyGIRIk52fmktkvghpG9g44irZQKQSQKbN51mOWbS7lrXDpJ8bFBx5FWSoUgEgXmZeeTFB/DHWP7BR1FWjEVgkjA9hwq59UPSvhqZh86JScEHUdaMRWCSMAWrC6ksqaGmRP0QTQJlgpBJEBHT1Tx3JrtTB7ag35dkoOOI62cCkEkQL/NKaLseKW+L1miggpBJCBV1TU8vaqA0f06Mapvp6DjiKgQRIKybMNuivYfZ1aWXh1IdFAhiASgdpqKraR3acu1Q7oHHUcEUCGIBGJt4QHyisuYmdWfWE1TIVFChSASgDkr8unUNp6bRqUFHUXkFBWCSBPbWnqENzft5s5x6bRJ0DQVEj1UCCJNbF52AQlxMfzdOE1TIdElokIws8lmttnMtpjZw2cZd5OZuZllhq7Hm9kCM1tvZpvM7HthY79lZhvM7CMze8HMki58c0Si294jJ1iUW8yMUWmktksMOo7IZ5yzEMwsFngKmAIMAW4zsyF1jEsBHgTeDVt8M5Do7sOB0cDXzCzdzHqHxma6+zAgFrj1QjdGJNr9avU2KqpquE/flyxRKJJXCGOALe6e7+4VwIvA9DrG/Rh4FCgPW+ZAspnFAW2ACuBQ6LY4oE3otrZAyfltgkjzcLyimufWbOOawd0Z0LVd0HFEPieSQugNFIVdLw4tO8XMRgJ93P3109ZdCBwFdgLbgZ+4+3533wH8JLRsJ1Dm7n+q68HNbLaZ5ZhZTmlpaSTbJBKVFuUWs/9ohb4vWaJWJIVQ10nSfupGsxjgMeA7dYwbA1QDvYAM4Dtm1t/MOlH7KiMjdFuymd1R14O7+xx3z3T3zK5du0YQVyT6VNc4T68sYERaBy5P1zQVEp0iKYRioE/Y9TQ+e3gnBRgGLDezQmAssDj0xvLtwFJ3r3T3PcAqIBO4Bihw91J3rwR+B1x5oRsjEq3e3LSbgr1HmTWxP2b6IJpEp0gKYS0w0MwyzCyB2jd/F5+80d3L3D3V3dPdPR1YA0xz9xxqDwl9yWolU1sWH4eWjzWztlb7v+NqYFODbplIFJm7Ip+0Tm2YPLRH0FFEzuicheDuVcADwDJqn7RfdvcNZvaImU07x+pPAe2Aj6gtlvnu/qG7v0vt+wu5wPpQjjnnvxki0WvdtgPkbDvAzAkZxMXqoz8Svczdzz0qSmRmZnpOTk7QMUTq5f7n1rFqy15Wf+9qkhPjgo4jrZCZrXP3zHON068rIo1o276jLN2wizvG9lMZSNRTIYg0oqdXFhAXY9x9ZXrQUUTOSYUg0kgOHK3g5ZwibrisN93aa2YWiX4qBJFG8tyabZRX1uj7kqXZUCGINILyymoWrC7kqkFdubh7StBxRCKiQhBpBK9+sIO9RyqYre9LlmZEhSDSwGpqnLnZBQzt1Z5xA7oEHUckYioEkQa2/JM9bNlzhFlZmqZCmhcVgkgDm7Min54dkrju0p5BRxGpFxWCSAP6sPgga/L3c+/4DOI1TYU0M/qJFWlAc7MLSEmM49Yxfc49WCTKqBBEGkjR/mMsWb+T267oS0pSfNBxROpNhSDSQOavKsRA01RIs6VCEGkAZccqeXHtdq4f0YteHdsEHUfkvKgQRBrA8+9t51hFNfdlZQQdReS8qRBELlBFVQ3zVxUw4aJUhvbqEHQckfOmCdqlRXN3qmqc6hqnsrqGqura61U1tZcrq2tCt4WW1XjtmOqaU+Mqq09f/6/jKqtr2Fp6hD2HT/BfN48IenNFLogKQT6nqrqGsuOVVNXU8YR58gm1OuxJMbS8OvTkebZx1dVO5WnL//qkG3qSrgm7rzrHhZ6cI3iir6ppmm8EvKxPRyYOTG2SxxJpLCoEAWp/k95QcoiF64pZnFfC/qMVjfp4cTFGXKwRHxNDbKwRFxNDfKwRG2PEx8YQFxN2OTQuLiaGpPjaZbW31a538r7iQuuF31dcbAzxMUbsyfv4zLjP39dnxsWEjQ3LGL7+yctt4mM1TYU0eyqEVm7PoXJe+WAHi9btYPPuwyTExnDNkG6MSe9MfFzMWZ9E42LDnlDr8SQaG2N68hSJQiqEVqi8spo3N+1m4bpiVnxSSo3XHvL48Q3DuP7SnnRsmxB0RBEJgAqhlXB3crcfZFFuMa/nlXCovIqeHZL4+hcGMGN0GgO6tgs6oogETIXQwu04eJzf5xazKHcHBXuP0iY+lsnDejBjVBrjBnQhNkaHbkSklgqhBTp6ooqlH+1iUW4xq/P34Q5XZHTm/qsGMHV4T9ol6p9dRD5PzwwtRE2NsyZ/Hwtzi1n60S6OVVTTr0tbHrr6Yr4yqjd9OrcNOqKIRDkVQjNXsPcoi9YV8/v3d7Dj4HFSEuOYNqIXM0ankdmvk87mEZGIqRCaobLjlbz+YQmL1hWTu/0gMQYTBnbl/04exKShPUiKjw06oog0QyqEZqKquobsT/eyMLeYNzbupqKqhoHd2vHwlEu4cWRvurdPCjqiiDRzKoQo9/GuQyxaV8wrH5RQevgEndrGc9vlfZgxOo3hvTvokJCINBgVQhTad+QEr35QwqLcYjaUHCIuxvjiJd2YMSqNL13SjYQ4TVIrIg1PhRAlKqpq+PPHu1m4bgfLN++hqsYZ1rs9P7x+CNNG9KJLu8SgI4pIC6dCCJC782FxGYtyayeUO3iskq4pidw7IYMZo9IY1CMl6Igi0oqoEAKwq6yc37+/g0W5xWzZc4SEuBj+Zkh3ZoxOI+uiVOJidUhIRJqeCqGJHK+o5k8bd7FwXTGrtuylxmF0v078243Due7SnnRoEx90RBFp5VQIjcjdWVt4gEXrilmyfieHT1TRu2MbvvHFi/jKqDQyUpODjigickpEhWBmk4EngFhgnrv/xxnG3QT8Frjc3XPMLB6YB4wKPdav3P3fzWwQ8FLYqv2BH7j74+e/KdGjaP8xFuUW87vcHWzff4y2CbFMGdaTGaN7MzajCzGaUE5EotA5C8HMYoGngGuBYmCtmS12942njUsBHgTeDVt8M5Do7sPNrC2w0cxecPfNwGVh978D+H1DbFBQjpyoYsmHO1mYW8x7Bfsxg3H9u/DNqwcyeVgPkjWhnIhEuUiepcYAW9w9H8DMXgSmAxtPG/dj4FHgu2HLHEg2szigDVABHDptvauBre6+rf7xg1Vd47yzdS+L1hWzdMMuyitryEhN5rt/czE3jkqjd8c2QUcUEYlYJIXQGygKu14MXBE+wMxGAn3c/XUzCy+EhdSWx06gLfAtd99/2v3fCrxQ3+BB2rLnCItyi3nl/R3sLCunfVIcXxmVxoxRaYzq21GfHhaRZimSQqjr2c1P3WgWAzwG3F3HuDFANdAL6ARkm9mbYa82EoBpwPfO+OBms4HZAH379o0gbuM4eKyC1/JKWJi7g7yig8TGGBMHpvL96wZzzeDumlBORJq9SAqhGOgTdj0NKAm7ngIMA5aHfjPuASw2s2nA7cBSd68E9pjZKiATyA+tOwXIdffdZ3pwd58DzAHIzMz0M41rDJXVNfxlcymLcot5a9MeKqpruKRHCt+fOpjpI3vRLUUTyolIyxFJIawFBppZBrVv/t5K7RM9AO5eBqSevG5my4HD0mC2AAAGNUlEQVTvhs4yuhr4kpk9R+0ho7FA+JlEtxGFh4s2lJSxaN0OFuftYO+RCrokJ/C3Y/syY1QaQ3u11yEhEWmRzlkI7l5lZg8Ay6g97fQZd99gZo8AOe6++CyrPwXMBz6i9tDTfHf/ECB01tG1wNcucBsaROnhE7z6wQ4Wrivm412HiY81rr6k9tPDVw3qSrw+PSwiLZy5N+lRmAuSmZnpOTk5DXZ/5ZXVvLVpD4tyi/nLJ6VU1zgj+nTkplG9+fKlveiUnNBgjyUiEhQzW+fumeca1+pOjnd33i86yKJ1xbyWV8Kh8iq6t09kVlZ/bhrdm4u6aUI5EWmdWk0hlBw8fmpCufzSoyTFxzBpaA9mjEpj/EWpxOrTwyLSyrX4QnB3Zi7I4e3Ne3CHMemd+drE/kwd3pOUJE0oJyJyUosvBDMjIzWZ4b0HMmNUGn27tA06kohIVGrxhQDwz18eEnQEEZGop3MpRUQEUCGIiEiICkFERAAVgoiIhKgQREQEUCGIiEiICkFERAAVgoiIhDSr2U7NrBQ43+9eTgX2NmCchqJc9aNc9aNc9dMSc+0FcPfJ5xrYrArhQphZTiTTvzY15aof5aof5aqf1p5Lh4xERARQIYiISEhrKoQ5QQc4A+WqH+WqH+Wqn1adq9W8hyAiImfXml4hiIjIWTT7QjCzyWa22cy2mNnDddx+t5mVmtkHoT/3hd12l5l9GvpzVxTlqg5bvrghc0WSLTTmq2a20cw2mNnzYcsD22fnyNVo+yyCf8vHwh77EzM7GHZbkD9jZ8sV5P7qa2Zvm9n7ZvahmU0Nu+17ofU2m9mkaMhlZulmdjxsf/2yiXP1M7O3QpmWm1la2G0N+/Pl7s32DxALbAX6AwlAHjDktDF3A/9dx7qdgfzQ351ClzsFnSt025GA99lA4P2T+wPoFiX7rM5cjbnPIsl12vi/B56Jhv11plxB7y9qj4ffH7o8BCgMu5wHJAIZofuJjYJc6cBHAe6v3wJ3hS5/Cfh1Y/18NfdXCGOALe6e7+4VwIvA9AjXnQS84e773f0A8AZwzg9uNEGuxhZJtlnAU6H9grvvCS0Pep+dKVdjqu+/5W3AC6HLQe+vM+VqTJHkcqB96HIHoCR0eTrworufcPcCYEvo/oLO1ZgiyTUEeCt0+e2w2xv856u5F0JvoCjsenFo2elmhF5uLTSzPvVct6lzASSZWY6ZrTGzGxooU32yXQxcbGarQhkm12PdIHJB4+2ziLfZzPpR+5vtn+u7bhPngmD314+AO8ysGFhC7auXSNcNIhdARuhQ0l/MLKuBMkWaKw+YEbp8I5BiZl0iXLdemnshWB3LTj9t6jUg3d0vBd4EFtRj3SByAfT12k8l3g48bmYDGihXpNniqD08cxW1v1nOM7OOEa4bRC5ovH1Wn22+FVjo7tXnsW59XUguCHZ/3QY86+5pwFTg12YWE+G6QeTaSe3+Ggl8G3jezNrTMCLJ9V3gC2b2PvAFYAdQFeG69dLcC6EYCP/NOo3TXua5+z53PxG6OhcYHem6AeXC3UtCf+cDy4GRDZQromyhMa+6e2Xopftmap+IA91nZ8nVmPusPtt8K589LBP0/jpTrqD310zg5dDjrwaSqJ2rJ+j9VWeu0CGsfaHl66g95n9xU+Vy9xJ3/0qokL4fWlYW4TbVT2O8UdJUf6j9jTGf2pfDJ9+QGXramJ5hl28E1vhf35ApoPbNmE6hy52jIFcnIDF0ORX4lLO8WdhI2SYDC8IyFAFdomCfnSlXo+2zSHKFxg0CCgl9ticafsbOkivQ/QX8Ebg7dHkwtU9iBgzls28q59NwbypfSK6uJ3NQ++bvjib+uU8FYkKX/xV4pLF+vi54g4L+Q+1Lu0+obe3vh5Y9AkwLXf53YENoR78NXBK27r3UvnG1BbgnGnIBVwLrQ8vXAzMD2GcG/AzYGMpwa5TsszpzNfY+O1eu0PUfAf9Rx7qB7a8z5Qp6f1H7Jumq0ON/APxN2LrfD623GZgSDbmoPX5/8v9qLnB9E+e6idrS/gSYR6jMG+PnS59UFhERoPm/hyAiIg1EhSAiIoAKQUREQlQIIiICqBBERCREhSAiIoAKQUREQlQIIiICwP8HgAkH8Pd/kwIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = str(max_depths[i]) +' Test')\n",
    "#plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = str(max_depths[i]) +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:07:32.351589Z",
     "start_time": "2020-06-27T11:07:32.348599Z"
    }
   },
   "source": [
    "### 继续细调colsample_bytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:58:10.922351Z",
     "start_time": "2020-06-27T11:54:46.231692Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  2.3min remaining:  1.2min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  3.4min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=1.0,\n",
       "        importance_type='split', learning_rate=0.1, max_bin=127,\n",
       "        max_depth=7, min_child_samples=30, min_child_weight=0.001,\n",
       "        min_split_gain=0.0, n_estimators=335, n_jobs=4, num_class=9,\n",
       "        num_leaves=60, objective='multiclass', random_state=None,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.1, 0.2, 0.3, 0.4]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':60,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(1,5)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:58:55.303551Z",
     "start_time": "2020-06-27T11:58:55.299562Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.486805491547\n",
      "{'colsample_bytree': 0.3}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T11:58:59.762981Z",
     "start_time": "2020-06-27T11:58:59.697082Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt4VfWd9/33NwnhmACBcMqBBAExHETZBDq21aFF0amkHsaK1zjtOK2292N9RntfFaszdlBHbeuj7VPvPrfjrZ3OfQvaghZRx9PIqFUhAYRwkIMEkwCGQDiHkNP3+WNv4pZGsslp7Z18Xte1L/da+/fb+/tz6f7kt9baa5m7IyIikhR0ASIiEh8UCCIiAigQREQkQoEgIiKAAkFERCIUCCIiAsQYCGY2z8y2mtkOM1vYyuvfMbNqM/sw8vhu1GvfNrPtkce3I+sGmNlLZvaRmW0ys4c6b0giItIe1tbvEMwsGdgGzAUqgWJggbtvjmrzHSDk7ree1jcDKAFCgANrgBnASWCWu79lZqnAm8C/uPsrnTQuERE5S7HMEAqBHe6+093rgSVAUYzvfxnwurvXuPtB4HVgnrvXuvtbAJH3XAtkn335IiLSWVJiaJMFVEQtVwKzWml3jZl9lfBs4nZ3r/iCvlnRncxsCHAl8Mu2Chk+fLjn5eXFULKIiJyyZs2a/e6e2Va7WALBWll3+n6mF4HF7n7SzL4P/Bswp62+ZpYCLAZ+5e47W/1ws5uBmwFyc3MpKSmJoWQRETnFzD6JpV0su4wqgZyo5WxgT3QDdz/g7icji/9K+DhBLH2fALa7+2Nf9OHu/oS7h9w9lJnZZsCJiEg7xRIIxcAEM8uPHAC+Hlge3cDMRkctzge2RJ6/ClxqZkPNbChwaWQdZnY/MBj4h44NQUREOkObu4zcvdHMbiX8RZ4MPOXum8xsEVDi7suB28xsPtAI1ADfifStMbP7CIcKwKLIumzgbuAjYK2ZAfza3Z/s3OGJiEis2jztNJ6EQiHXMQQRkbNjZmvcPdRWO/1SWUREAAWCiIhEKBBERAToBYHg7jxXXMFrmz4NuhQRkbgWyw/TElpTs/PvH3zC3sN1FOZnMGRAatAliYjEpR4/Q0hJTuLha6ZxqLae+1/a0nYHEZFeqscHAkDBmHRuuXgcf1hTyTvbq4MuR0QkLvWKQAD44ZwJjMscyF3LSqmtbwy6HBGRuNNrAqFfn2QeunoalQdP8Mhr24IuR0Qk7vSaQAAozM/gb2bn8vSfylhXfjDockRE4kqvCgSAO+dNYmR6PxYuLaW+sTnockRE4kavC4S0fn144KopbK06ym9Wfhx0OSIicaPXBQLAnEkjmX/+GH791na2Vx0NuhwRkbjQKwMB4N4rCxjUN4U7l26gqTlxrvgqItJVem0gDBvUl3+6soC15Yf49/d3BV2OiEjgem0gAHxzehaXnJvJz17dSuXB2qDLEREJVK8OBDPj/m9OAeAnz28kkW4WJCLS2Xp1IABkDx3AnfMm8fa2ap5ftzvockREAtPrAwHgxtljmTF2KItWbGb/sZNBlyMiEoiYAsHM5pnZVjPbYWYLW3n9O2ZWbWYfRh7fjXrt22a2PfL4dtT6GWZWGnnPX5mZdc6Qzl5SkvHwNVOpPdnEP7+4OagyREQC1WYgmFky8DhwOVAALDCzglaaPuvu0yOPJyN9M4B7gVlAIXCvmQ2NtP8NcDMwIfKY19HBdMT4EWncOmc8L67fw5tbqoIsRUQkELHMEAqBHe6+093rgSVAUYzvfxnwurvXuPtB4HVgnpmNBtLd/X0PH8n9HfDNdtTfqb5/8TmcOzKNu5/fyNG6hqDLERHpVrEEQhZQEbVcGVl3umvMbIOZ/cHMctromxV53tZ7YmY3m1mJmZVUV3ftvQxSU5J4+Npp7Dtax0OvfNSlnyUiEm9iCYTW9u2ffn7mi0Ceu08D3gD+rY2+sbxneKX7E+4ecvdQZmZmDOV2zPScIdx0UT7/Z1U5q3Ye6PLPExGJF7EEQiWQE7WcDeyJbuDuB9z91Ok5/wrMaKNvZeT5F75nkO64dCI5Gf25a1kpdQ1NQZcjItItYgmEYmCCmeWbWSpwPbA8ukHkmMAp84FTNy9+FbjUzIZGDiZfCrzq7nuBo2Y2O3J20d8Cf+zgWDrNgNQUHrxqGjv3H+dXb24PuhwRkW7RZiC4eyNwK+Ev9y3Ac+6+ycwWmdn8SLPbzGyTma0HbgO+E+lbA9xHOFSKgUWRdQA/AJ4EdgAfA6902qg6wZcnDOevZ2TzP9/eyaY9h4MuR0Sky1kiXa4hFAp5SUlJt33e4doGvvb//BejBvflhf92ESnJ+h2fiCQeM1vj7qG22ukb7gwGD+jDoqLJbNx9hCffLQu6HBGRLqVAaMPlU0Zx2eSRPPr6Nsr2Hw+6HBGRLqNAaIOZsahoCqkpSdy1bIOuiCoiPZYCIQYj0/tx9xXn8cHOGpYUV7TdQUQkASkQYvStmTl8adww/uXlLVQdqQu6HBGRTqdAiJGZ8eDVU6lvbOaeF3QzHRHpeRQIZyFv+EB+dOlEXt9cxculnwZdjohIp1IgnKWbLspnatZg7l2+kUO19UGXIyLSaRQIZyklOYmHr5nGodoG7n9pS9sdREQShAKhHQrGpHPLxeP4w5pK3tnetZfkFhHpLgqEdvrhnAmMyxzIXctKqa1vDLocEZEOUyC0U78+yTx09TQqD57gF69uC7ocEZEOUyB0QGF+BjfOHsvT75Wxrvxg0OWIiHSIAqGDfjzvXEal92Ph0lLqG5uDLkdEpN0UCB2U1q8PD1w1ha1VR/nNyo+DLkdEpN0UCJ1gzqSRzD9/DL9+azvbq44GXY6ISLsoEDrJvVcWMKhvCncu3UBTsy5rISKJR4HQSYYN6ss/XVnA2vJD/O79XUGXIyJy1mIKBDObZ2ZbzWyHmS08Q7trzczNLBRZTjWzp82s1MzWm9klUW0XRNZvMLP/MLPhHR5NwL45PYtLzs3k569upfJgbdDliIiclTYDwcySgceBy4ECYIGZFbTSLg24DVgVtfp7AO4+FZgLPGJmSWaWAvwS+Et3nwZsAG7t4FgCZ2Y8cNVUDPjJ87oiqogkllhmCIXADnff6e71wBKgqJV29wE/A6JvFlAAvAng7vuAQ0AIsMhjoJkZkA7sae8g4knWkP78eN4k3t5WzfPrdgddjohIzGIJhCwg+jZhlZF1LczsAiDH3Vec1nc9UGRmKWaWD8yItGsAfgCUEg6CAuB/tfbhZnazmZWYWUl1dWJcN+jG2WOZMXYoi1ZsZv+xk0GXIyISk1gCwVpZ17IvxMySgEeBH7XS7inCAVICPAa8BzSaWR/CgXABMIbwLqO7Wvtwd3/C3UPuHsrMzIyh3OAlJRkPXzOV2pNN/POLm4MuR0QkJrEEQiWQE7Wczed376QBU4CVZrYLmA0sN7OQuze6++3uPt3di4AhwHZgOoC7f+zhHe3PAX/R4dHEkfEj0rh1znheXL+HNzZXBV2OiEibYgmEYmCCmeWbWSpwPbD81Ivuftjdh7t7nrvnAR8A8929xMwGmNlAADObCzS6+2ZgN1BgZqf+5J8L9LibC3z/4nOYNCqNe17YyJG6hqDLERE5ozYDwd0bCZ8B9CrhL+3n3H2TmS0ys/ltdB8BrDWzLcCdwI2R99wD/DPwtpltIDxj+Jf2DyM+paaEb6az72gdD7/yUdDliIickSXSqZGhUMhLSkqCLuOs3b9iM0++W8azN89m1rhhQZcjIr2Mma1x91Bb7fRL5W5wx6UTycnoz13LSqlraAq6HBGRVikQusGA1BQevGoaO/cf51dvbg+6HBGRVikQusmXJwznr2dk8z/f3snG3YeDLkdE5M8oELrRPX9VQMbAVBYu20Bjk26mIyLxRYHQjQYP6MOi+ZPZuPsIT75bFnQ5IiKfo0DoZpdPHc1lk0fy6OvbKNt/POhyRERaKBACsKhoCqkpSdy1bIOuiCoicUOBEICR6f24+4rz+GBnDUuKK9ruICLSDRQIAfnWzBy+NG4Y//LSFj49XNd2BxGRLqZACIiZ8eDVU2lobuYf/6ib6YhI8BQIAcobPpA75k7k9c1VvFz6adDliEgvp0AI2E0X5TM1azD3Lt/Iodr6oMsRkV5MgRCwlOTwFVEP1TZw/0s97grgIpJAFAhxoGBMOrdcPI4/rKnkne2JcZtQEel5FAhx4odzJjAucyB3LSvl+MnGoMsRkV5IgRAn+vVJ5uFrplF58ASPvLYt6HJEpBdSIMSRmXkZ3Dh7LE+/V8a68oNBlyMivYwCIc78eN65jErvx8KlpdQ36oqoItJ9YgoEM5tnZlvNbIeZLTxDu2vNzM0sFFlONbOnzazUzNab2SVRbVPN7Akz22ZmH5nZNR0eTQ+Q1q8PD1w1ha1VR/nNyo+DLkdEepE2A8HMkoHHgcuBAmCBmRW00i4NuA1YFbX6ewDuPhWYCzxiZqc+825gn7tPjLzvf3VgHD3KnEkjmX/+GH791na2VR0NuhwR6SVimSEUAjvcfae71wNLgKJW2t0H/AyIvjBPAfAmgLvvAw4Bp270fBPwYOS1Znff364R9FD3XlnAoL4p3Ll0A03NuqyFiHS9WAIhC4i+JGdlZF0LM7sAyHH3Faf1XQ8UmVmKmeUDM4AcMxsSef0+M1trZr83s5HtG0LPNGxQX+69cjLryg/xu/d3BV2OiPQCsQSCtbKu5U/WyC6gR4EftdLuKcIBUgI8BrwHNAIpQDbwJ3e/EHgf+EWrH252s5mVmFlJdXXv+tFW0fQxXHJuJj9/dSuVB2uDLkdEerhYAqESyIlazgb2RC2nAVOAlWa2C5gNLDezkLs3uvvt7j7d3YuAIcB24ABQCzwfeY/fAxe29uHu/oS7h9w9lJmZeRZDS3xmxgNXTcWAnzyvK6KKSNeKJRCKgQlmlm9mqcD1wPJTL7r7YXcf7u557p4HfADMd/cSMxtgZgMBzGwu0Ojumz38zfYicEnkbb4GbO60UfUgWUP68+N5k3h7WzXPr9sddDki0oO1GQju3gjcCrwKbAGec/dNZrbIzOa30X0EsNbMtgB3AjdGvXYn8FMz2xBZ39ouJwFunD2WGWOHsmjFZvYfOxl0OSLSQ1ki7YYIhUJeUlISdBmB2LHvKFf88l0unTySX9/Q6t41EZFWmdkadw+11U6/VE4Q40ek8cM541mxYS9vbK4KuhwR6YEUCAnklovPYdKoNO55YSNH6hqCLkdEehgFQgJJTQnfTGff0ToefuWjoMsRkR5GgZBgzs8Zwk0X5fN/VpWzaueBoMsRkR5EgZCA7rh0IjkZ/Vm4rJS6hqagyxGRHkKBkIAGpKbw4FXTKNt/nF++uT3ockSkh1AgJKgvTxjOdaFsnnh7Jxt3Hw66HBHpARQICezuKwrIGJjKwmUbaGzSzXREpGMUCAls8IA+LJo/mY27j/Dku2VBlyMiCU6BkOAunzqayyaP5NHXt1G2/3jQ5YhIAlMg9ACLiqaQmpLEwqUbaNbNdESknRQIPcDI9H7cfcV5rCqrYUlxRdsdRERaoUDoIb41M4cvjRvGgy9v4dPDdW13EBE5jQKhhzAzHrpmKg3NzfzjH3UzHRE5ewqEHmTssIHcMXcir2+u4uXST4MuR0QSjAKhh7nponymZg3m3uUbOVRbH3Q5IpJAFAg9TEpy+Iqoh2obuG/FlqDLEZEEokDogQrGpHPLxeNYuraSt7dVB12OiCSImALBzOaZ2VYz22FmC8/Q7lozczMLRZZTzexpMys1s/VmdkkrfZab2cZ2j0Ba9cM5ExiXOZCfPF/K8ZONQZcjIgmgzUAws2TgceByoABYYGYFrbRLA24DVkWt/h6Au08F5gKPmFlSVJ+rgWMdGYC0rl+fZB6+ZhqVB0/wyGvbgi5HRBJALDOEQmCHu+9093pgCVDUSrv7gJ8B0SfBFwBvArj7PuAQcGr2MAi4A7i/3dXLGc3My+DG2WN5+r0y1pYfDLocEYlzsQRCFhD989fKyLoWZnYBkOPuK07rux4oMrMUM8sHZgA5kdfuAx4BattTuMTmx/POZVR6PxYu3UB9o66IKiJfLJZAsFbWtfzqKbIL6FHgR620e4pwgJQAjwHvAY1mNh0Y7+7Pt/nhZjebWYmZlVRX6wDp2Urr14cHrprCtqpj/I+VO4IuR0TiWCyBUMlnf9UDZAN7opbTgCnASjPbBcwGlptZyN0b3f12d5/u7kXAEGA78CVgRqT9u8BEM1vZ2oe7+xPuHnL3UGZm5tmNTgCYM2kk888fw+Nv7WBb1dGgyxGROBVLIBQDE8ws38xSgeuB5adedPfD7j7c3fPcPQ/4AJjv7iVmNsDMBgKY2Vyg0d03u/tv3H1MpP2XgW3ufknnDk2i3XtlAYP6pnDn0g006YqoItKKNgPB3RuBW4FXgS3Ac+6+ycwWmdn8NrqPANaa2RbgTuDGjhYs7TNsUF/uvXIy68oP8bv3dwVdjojEIUuki6CFQiEvKSkJuoyE5e783W+LWV1Ww6v/8FVyMgYEXZKIdAMzW+Puobba6ZfKvYiZ8cBVUzHgJ8+X6oqoIvI5CoReJmtIf348bxLvbN/PsrW7gy5HROKIAqEXunH2WGaMHcp9L21m/7GTQZcjInFCgdALJSUZD18zldqTTfx0+aagyxGROKFA6KXGj0jjh3PGs2LDXt7YXBV0OSISBxQIvdgtF5/DpFFp3PPCRo7UNQRdjogETIHQi6WmhG+ms+9oHQ+98lHQ5YhIwBQIvdz5OUO46aJ8nllVzgc7DwRdjogESIEg3HHpRHIy+nPXslLqGpqCLkdEAqJAEAakpvDQ1dMo23+cX765PehyRCQgCgQB4KLxw7kulM0Tb+9k4+7DQZcjIgFQIEiLu68oIGNgKncu3UBjk26mI9LbKBCkxeABfVg0fzKb9hzhX98pC7ocEelmCgT5nMunjuayySN57I1tlO0/HnQ5ItKNFAjyZ+4rmkJqShILl26gWTfTEek1FAjyZ0ak9+OevzqPVWU1LCmuCLocEekmCgRp1XWhHP7inGE8+PIWPj1cF3Q5ItINFAjSKjPjwaun0tDczD0vbNTNdER6AQWCfKGxwwZyx9yJvLGlipdK9wZdjoh0sZgCwczmmdlWM9thZgvP0O5aM3MzC0WWU83saTMrNbP1ZnZJZP0AM3vJzD4ys01m9lCnjEY63U0X5TM1azA/Xb6Jg8frgy5HRLpQm4FgZsnA48DlQAGwwMwKWmmXBtwGrIpa/T0Ad58KzAUeMbNTn/kLd58EXABcZGaXd2Qg0jVSksNXRD1U28D9L20JuhwR6UKxzBAKgR3uvtPd64ElQFEr7e4DfgZEH4EsAN4EcPd9wCEg5O617v5WZH09sBbIbvcopEsVjEnn+xefw9K1lby9rTrockSki8QSCFlA9LmHlZF1LczsAiDH3Vec1nc9UGRmKWaWD8wAck7rOwS4kkhwnM7MbjazEjMrqa7Wl1FQbp0znnGZA/nJ86UcP9kYdDki0gViCQRrZV3LKSeRXUCPAj9qpd1ThAOkBHgMeA9ojOqbAiwGfuXuO1v7cHd/wt1D7h7KzMyMoVzpCv36JPPwNdOoPHiCX7y2NehyRKQLxBIIlXz+r/psYE/UchowBVhpZruA2cByMwu5e6O73+7u0929CBgCRF9f+Qlgu7s/1pFBSPeYmZfBjbPH8tv3drG2/GDQ5YhIJ4slEIqBCWaWb2apwPXA8lMvuvthdx/u7nnungd8AMx395LI2UQDAcxsLtDo7psjy/cDg4F/6NwhSVf68bxzGZXej4VLN1DfqCuiivQkbQaCuzcCtwKvAluA59x9k5ktMrP5bXQfAaw1sy3AncCNAGaWDdxN+KDzWjP70My+24FxSDdJ69eHB66awraqY/yPlTuCLkdEOpEl0i9QQ6GQl5SUBF2GAP/3knW8XLqXl277ChNHpgVdjoicgZmtcfdQW+30S2Vpl3/6RgGD+qZw59INNOmKqCI9ggJB2mXYoL7ce+Vk1pUf4t/e2xV0OSLSCRQI0m5F08dwybmZ/PzVrVTU1AZdjoh0kAJB2s3MeOCqqSQZ/OT5Ul0RVSTBKRCkQ7KG9OfOyyfxzvb9LFu7O+hyRKQDFAjSYX8zayyhsUO576XN7D92MuhyRKSdFAjSYUlJxkPXTKP2ZBM/Xb4p6HJEpJ0UCNIpxo8YxA/njGfFhr28vrkq6HJEpB0UCNJpbrn4HCaNSuOeF0o5UtcQdDkicpYUCNJpUlPCN9OpPnqSh175KOhyROQsKRCkU52fM4S//3I+z6wq54OdB4IuR0TOggJBOt0dc88lN2MAdy0rpa6hKehyRCRGCgTpdP1Tk3nw6qmU7T/OL9/c3nYHEYkLCgTpEheNH851oWyeeHsnG3cfDrocEYmBAkG6zN1XFJAxMJU7l26gsUk30xGJdwoE6TKDB/Rh0fzJbNpzhH99pyzockSkDQoE6VKXTx3NvMmjeOyNbZTtPx50OSJyBgoE6XKLiibTNyWJhUs30Kyb6YjErZgCwczmmdlWM9thZgvP0O5aM3MzC0WWU83saTMrNbP1ZnZJVNsZkfU7zOxXZmYdHo3EpRHp/bj7r85jVVkNS4orgi5HRL5Am4FgZsnA48DlQAGwwMwKWmmXBtwGrIpa/T0Ad58KzAUeMbNTn/kb4GZgQuQxr/3DkHh3XSiHvzhnGA++vIVPD9cFXY6ItCKWGUIhsMPdd7p7PbAEKGql3X3Az4Do/9sLgDcB3H0fcAgImdloIN3d3/fwXVV+B3yz/cOQeGdmPHj1VBqam7nnhY26mY5IHIolELKA6Hl+ZWRdCzO7AMhx9xWn9V0PFJlZipnlAzOAnEj/yjO9Z9R732xmJWZWUl1dHUO5Eq/GDhvIHXMn8saWKl4q3Rt0OSJymlgCobV9+y1/3kV2AT0K/KiVdk8R/rIvAR4D3gMa23rPz610f8LdQ+4eyszMjKFciWc3XZTPtOzB/HT5Jg4erw+6HBGJEksgVBL+q/6UbGBP1HIaMAVYaWa7gNnAcjMLuXuju9/u7tPdvQgYAmyPvGf2Gd5TeqiU5PAVUQ/VNnDfis0660gkjsQSCMXABDPLN7NU4Hpg+akX3f2wuw939zx3zwM+AOa7e4mZDTCzgQBmNhdodPfN7r4XOGpmsyNnF/0t8MdOHpvEqfNGp/ODS85h2brdXPyLt3j8rR3sO6oDzSJBS2mrgbs3mtmtwKtAMvCUu28ys0VAibsvP0P3EcCrZtYM7AZujHrtB8Bvgf7AK5GH9BK3f30iE0amsXhVOT9/dSuPvr6Nr583kgWzcvnK+OEkJeksZJHuZol0tkcoFPKSkpKgy5BOtrP6GM8WV/D7NZXUHK8ne2h/rp+Zw1+HchiZ3i/o8kQSnpmtcfdQm+0UCBIvTjY28frmKhavLudPOw6QnGR8bdIIFszK5asTMknWrEGkXWINhDZ3GYl0l74pyXxj2hi+MW0Mu/YfZ0lxBX9YU8Frm6vIGtKfb83M4bpQDqMGa9Yg0hU0Q5C4Vt/YzBtbwrOGd7bvJ8lgzqSR3DArh4snjtCsQSQGmiFIj5CaksQVU0dzxdTRfHLgOM8WV/BcSSVvbKlizOB+XBeZNYwZ0j/oUkUSnmYIknAampp5c0sVz6yu4J3t1Rjwl+eOYEFhLpecm0lKsi7iKxJNB5WlV6ioqeXZ4gqeLamg+uhJRqWHZw3fmplDlmYNIoACQXqZhqZm/vOjfTyzqpy3t4eveXXJxEwWFOYyZ9IIzRqkV1MgSK9VUVPLcyUVPFdSQdWRk4xM78t1ofCxhpyMAUGXJ9LtFAjS6zU2NfPW1mqeWfUJK7eFZw1fnRCeNXztvBH00axBegkFgkiU3YdOhM9QKq7g0yN1ZKb15bpQNtfPzNWsQXo8BYJIKxqbmlm5tZrFq8t5a+s+HPjKhEwWzMzh6wUjNWuQHkmBINKGPYdO8FxJBc8WV7D3cB3DB/Xlr0PZLJiZS+4wzRqk51AgiMSoqdn5r237eGZVBf/5URXNDl+ZMJzrZ+Yyt2AkqSmaNUhw9hw6weY9R/h6wch2v4cCQaQdPj1c1zJr2H3oBMMGpnJtZNaQN3xg0OVJD+fulO0/TvGuGlaV1bC6rIbKgycwg3X/OJchA1Lb9b4KBJEOaGp23t5ezeJV5bz50T6amp2Lxg9jQWEulxaM0qxBOkVzs/PRp0dZXXaA4l0HWVVWw/5jJwEYNjCVwvwMZuZlUJifwXmj09t97S4FgkgnqTpSx+9LKli8OmrWMCObb83MYVzmoKDLkwRS39jMxj2HWR35679kVw1H6hoByBrSn8L8jJYQOCdzIOEbSnacAkGkkzU3O+/s2M/iVeW8saWKxmbnS+OGsWBWLpdNHknflOSgS5Q4c6K+iXUVB1sCYG35QeoamgE4J3Pg5wIge2jXncigQBDpQvuO1PH7NZUsKS6nouYEQwf04doZ2VxfmMs5mjX0WodPNLDmk/D+/+KyGkp3H6ahyTGDgtHp4QDIyyCUl0FmWt9uq6tTA8HM5gG/JHxP5Sfd/aEvaHct8HtgpruXmFkf4EngQsKX2v6duz8YaXs78F3AgVLg79z9jHdaVyBIvGludv708X4Wry7ntU3hWcOs/AxumJXLZZNH0a+PZg09WfXRkxTvCv/1v6qsho8+PYI79Ek2pmUPaZkBzBg7lPR+fQKrs9MCwcySgW3AXKASKAYWuPvm09qlAS8BqcCtkUC4AZjv7teb2QBgM3AJ0AC8CxS4+wkzew542d1/e6ZaFAgSz6qPnuQPkVnDJwdqGTKgD9dcmM2CwhzGj0gLujzpIHen8uCJlgBYXVbDzv3HAejfJ5kZY4e2HAC+IHdIXP0x0Jk3yCkEdrj7zsgbLwGKCH+5R7sP+Bnw36PWOTDQzFKA/kA9cCTyPAXob2YNwABgTwy1iMStzLS+/OCSc7jlq+N4f+cBnlldzu/e38X/ereMwrwMFszK4fIpo+Pqi0K+mLvzcfWxlt0/q8tq2HM4vBMjvV8KhfkZXF+Yw8y8DKZkDe4Rv3KPJRCygIqo5UpgVnQDM7sAyHH3FWYWHQhVdd6WAAAKOklEQVR/IBweewl/6d/u7jWRPr8AyoETwGvu/lq7RyESR5KSjIvGD+ei8cPZf+wkS9dUsnh1Obc/u56fLt/M1RdmsaAwl4kjNWuIJ03Nzpa9RyLn/4dPA605Xg+Ew74wP4PvR3YBTRyRRlIPvH1rLIHQ2qhb9jOZWRLwKPCdVtoVAk3AGGAo8I6ZvQEcJBwU+cAh4Pdm9jfu/r//7MPNbgZuBsjNzY2hXJH4MXxQX265+BxujswaFq+u4H9/8AlP/2kXobFDWVCYy19N06whCCcbm9hQ+dkpoGs+Ocixk+FTQHMzBvCX545gViQAxg4b0GmngMazWI4hfAn4qbtfFlm+CyDq4PBg4GPgWKTLKKAGmA/8HfCBu/97pO1TwH8QDpR57v73kfV/C8x29/92plp0DEF6ggPHTrJs7W4Wry5n5/7jpPdL4eoLs1lQmMu5ozRr6CrHTzaytvyzU0DXVRyivjF8CujEkYM+9yOw0YN71t32OvOgcgrhg8pfA3YTPqh8g7tv+oL2K4H/HjmofCcwCbiJ8C6jYuB6wscQngJmEt5l9FugxN3/3zPVokCQnsTdWVVWw+LV5bxS+in1Tc1cmDuEBYW5fGPaGPqnatbQEYdq6ynedZDVZQdYXVbDxj1HaGp2kpOMyWPSKcz77DcAQwe275IQiaKzTzu9AniM8GmnT7n7A2a2iPCX+PLT2q7ks0AYBDwNFBDe9fS0u/880u6fgW8BjcA64LvufvJMdSgQpKeqOV7PsrXhYw0fVx8nrV8KV10QPtZw3uj0oMtLCFVH6j53AHhr1VEAUlOSmJ4zpCUALhw7lEF9Y9lb3nPoh2kiCcjdKd51kMWry3mpdC/1jc1MzxnCDYW5fOP80QxI7V1fZF/E3SmvqW25AFzxrho+OVALwMDUZGbkZTAr8tf/tOzBvf4YjQJBJMEdqq1n2drdPLO6nB37jpHWN4WiC8awoDCXyWMGB11et2pudrbtO0px5AdgxbtqqDoS3qEwdECfln3/s/KHcd7oNFJ6wCmgnUmBINJDuDtrPjnIM6vLeWnDXk42NnN+9mAWFOZy5fljGNgDd380NDWzac+RyP7/gxTvquHwiQYARqX3Y9a48F//s/IzOCdzUI88BbQzKRBEeqDDtQ08v66SZ1aXs63qGANTkym6IIsbCnOZkpW4s4a6hiY+rDjUsvtnzScHqa1vAiB/+MCW/f+F+RlkD+3fK04B7UwKBJEezN1ZW36IxavLWbFhD3UNzUzNCs8a5k8fE/cHTY/WNbDmk89OAd1QeZj6pmbMYNKodArzhlKYP4yZ+UMZkdYv6HITngJBpJc4fKKBP364m2dWlfPRp0cZkJpM0fTwsYZp2UOCLg8I//YifA2gg6zedYDNe47Q7JCSZEzNHvzZVUDHZjB4QHAXgeupFAgivYy782FFeNbw4vq9nGhoYvKYdBYU5lI0fQxp3Xi1zT2HTrRcAbR4Vw079oV/t9o3JYkLc4dGDgBnMD13iM6c6gYKBJFe7EhdA3/8cA/PrCpny94j9O+TzPzzx7BgVi7nZw/u1H3w7s7O/cdbzv9fVVbD7kMnAEjrl0JobHj3T2F+BlOzBuv2owFQIIgI7s6GysMsXl3O8vV7qK1vomB0OgtmhWcN7blGf1Oz89GnR1oOAK8uq2H/sfBF4IYPSm3Z/TMzP4NJo9p/H2DpPAoEEfmco3UNLF8fnjVs2hOeNVx5/mgWFOYyPWfIF84a6hubKd196iJwByj55CBHo+4DfOoCcIX5GeQP77z7AEvnUSCIyBcqrTzMM6vLWf7hbo7XNzFpVBo3zMqlaHoWqclJrCs/2PIr4HUVn90HePyIQS3n/8/MzyBrSM+6CFxPpUAQkTYdO9nIi+v3sHh1ORsqD9M3JYmmZqex2UkyKBiTTmHeMArzhxLKy2D4oO67D7B0ns68Y5qI9FCD+qawoDCXBYW5bNx9mGVrd9O3TxKzIvcB7s4zkyR4CgQRAWBK1uCE/rWzdJzO/xIREUCBICIiEQoEEREBFAgiIhKhQBAREUCBICIiEQoEEREBFAgiIhKRUJeuMLNq4JN2dh8O7O/EcoLUU8bSU8YBGku86ilj6eg4xrp7ZluNEioQOsLMSmK5lkci6Clj6SnjAI0lXvWUsXTXOLTLSEREAAWCiIhE9KZAeCLoAjpRTxlLTxkHaCzxqqeMpVvG0WuOIYiIyJn1phmCiIicQcIHgpnNM7OtZrbDzBa28vpXzWytmTWa2bWnvfZtM9seeXy7+6puXQfH0mRmH0Yey7uv6tbFMJY7zGyzmW0wszfNbGzUa4m2Xc40lrjZLjGM4/tmVhqp9V0zK4h67a5Iv61mdln3Vv7n2jsWM8szsxNR2+T/6/7q/6zWM44lqt21ZuZmFopa17nbxd0T9gEkAx8D44BUYD1QcFqbPGAa8Dvg2qj1GcDOyD+HRp4PTcSxRF47FvT2OMux/CUwIPL8B8CzCbxdWh1LPG2XGMeRHvV8PvAfkecFkfZ9gfzI+yQn6FjygI1Bb4+zGUukXRrwNvABEOqq7ZLoM4RCYIe773T3emAJUBTdwN13ufsGoPm0vpcBr7t7jbsfBF4H5nVH0V+gI2OJN7GM5S13r40sfgBkR54n4nb5orHEk1jGcSRqcSBw6gBjEbDE3U+6exmwI/J+QenIWOJNm2OJuA/4GVAXta7Tt0uiB0IWUBG1XBlZ19V9u0JH6+lnZiVm9oGZfbNzSztrZzuWvwdeaWffrtaRsUD8bJeYxmFm/5eZfUz4y+e2s+nbjToyFoB8M1tnZv9lZl/p2lLb1OZYzOwCIMfdV5xt37OV6PdUtlbWxfqXQEf6doWO1pPr7nvMbBzwn2ZW6u4fd1JtZyvmsZjZ3wAh4OKz7dtNOjIWiJ/tEtM43P1x4HEzuwG4B/h2rH27UUfGspfwNjlgZjOAF8xs8mkziu50xrGYWRLwKPCds+3bHok+Q6gEcqKWs4E93dC3K3SoHnffE/nnTmAlcEFnFneWYhqLmX0duBuY7+4nz6ZvN+rIWOJpu5ztv9clwKkZTUJukygtY4nsXjkQeb6G8H73iV1UZyzaGksaMAVYaWa7gNnA8siB5c7fLkEfVOngAZkUwgcd8/nsgMzkL2j7W/78oHIZ4QOXQyPPMxJ0LEOBvpHnw4HttHJgKp7GQviL8WNgwmnrE267nGEscbNdYhzHhKjnVwIlkeeT+fzBy50Ee1C5I2PJPFU74QO5u+P9v6/T2q/ks4PKnb5dAvmX0Mn/Qq8AtkX+h7w7sm4R4b/UAGYSTtLjwAFgU1TfmwgfiNkB/F2ijgX4C6A08h9HKfD3CTCWN4Aq4MPIY3kCb5dWxxJv2yWGcfwS2BQZw1vRX0yEZz8fA1uByxNgm7Q6FuCayPr1wFrgyngfy2ltVxIJhK7YLvqlsoiIAIl/DEFERDqJAkFERAAFgoiIRCgQREQEUCCIiEiEAkFERAAFgoiIRCgQREQEgP8fxaB4JZ0+aosAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 减小学习率，调整n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:12:44.286834Z",
     "start_time": "2020-06-27T12:04:44.898050Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 3113\n",
      "best cv score: 0.472974886241\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          #'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':60,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.3,\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用所有训练数据，采用最佳参数重新训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:16:30.772960Z",
     "start_time": "2020-06-27T12:12:59.831097Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=0.3,\n",
       "        importance_type='split', learning_rate=0.01, max_bin=127,\n",
       "        max_depth=7, min_child_samples=30, min_child_weight=0.001,\n",
       "        min_split_gain=0.0, n_estimators=3113, n_jobs=4, num_class=9,\n",
       "        num_leaves=70, objective='multiclass', random_state=None,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "        subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          'n_estimators':n_estimators_2,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.3,\n",
    "         }\n",
    "\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "lg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:28:46.591920Z",
     "start_time": "2020-06-27T12:28:41.312944Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle as pk\n",
    "\n",
    "pk.dump(lg, open(\"Otto_LightGBM_goss_org_tfidf.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型测试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据准备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:29:13.651192Z",
     "start_time": "2020-06-27T12:28:56.492317Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1</th>\n",
       "      <th>feat_2</th>\n",
       "      <th>feat_3</th>\n",
       "      <th>feat_4</th>\n",
       "      <th>feat_5</th>\n",
       "      <th>feat_6</th>\n",
       "      <th>feat_7</th>\n",
       "      <th>feat_8</th>\n",
       "      <th>feat_9</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.421803</td>\n",
       "      <td>0.052224</td>\n",
       "      <td>0.842245</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.032787</td>\n",
       "      <td>0.039216</td>\n",
       "      <td>0.21875</td>\n",
       "      <td>0.228571</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.143963</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.070171</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.019608</td>\n",
       "      <td>0.18750</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078248</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.071995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.139311</td>\n",
       "      <td>0.034257</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.026316</td>\n",
       "      <td>0.026316</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.556178</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 187 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id    feat_1    feat_2   feat_3    feat_4  feat_5  feat_6    feat_7  \\\n",
       "0   1  0.000000  0.000000  0.00000  0.000000     0.0     0.0  0.000000   \n",
       "1   2  0.032787  0.039216  0.21875  0.228571     0.0     0.0  0.000000   \n",
       "2   3  0.000000  0.019608  0.18750  0.014286     0.0     0.0  0.000000   \n",
       "3   4  0.000000  0.000000  0.00000  0.014286     0.0     0.0  0.000000   \n",
       "4   5  0.016393  0.000000  0.00000  0.014286     0.0     0.0  0.026316   \n",
       "\n",
       "     feat_8  feat_9      ...        feat_84_tfidf  feat_85_tfidf  \\\n",
       "0  0.000000     0.0      ...                  0.0       0.000000   \n",
       "1  0.000000     0.0      ...                  0.0       0.000000   \n",
       "2  0.000000     0.0      ...                  0.0       0.000000   \n",
       "3  0.000000     0.0      ...                  0.0       0.139311   \n",
       "4  0.026316     0.0      ...                  0.0       0.000000   \n",
       "\n",
       "   feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  feat_90_tfidf  \\\n",
       "0       0.421803       0.052224       0.842245       0.000000            0.0   \n",
       "1       0.000000       0.000000       0.000000       0.143963            0.0   \n",
       "2       0.000000       0.000000       0.078248       0.000000            0.0   \n",
       "3       0.034257       0.000000       0.000000       0.000000            0.0   \n",
       "4       0.000000       0.000000       0.000000       0.000000            0.0   \n",
       "\n",
       "   feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0       0.000000       0.000000       0.000000  \n",
       "1       0.000000       0.070171       0.000000  \n",
       "2       0.000000       0.000000       0.071995  \n",
       "3       0.000000       0.000000       0.000000  \n",
       "4       0.556178       0.000000       0.000000  \n",
       "\n",
       "[5 rows x 187 columns]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test1 = pd.read_csv(\"Otto_FE_test_org_goss.csv\")\n",
    "test2 = pd.read_csv(\"Otto_FE_test_tfidf_goss.csv\")\n",
    "\n",
    "#去掉多余的id\n",
    "test2 = test2.drop([\"id\"], axis=1)\n",
    "test =  pd.concat([test1, test2], axis = 1, ignore_index=False)\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:29:20.299010Z",
     "start_time": "2020-06-27T12:29:19.142630Z"
    }
   },
   "outputs": [],
   "source": [
    "test_id = test['id']   \n",
    "X_test = test.drop([\"id\"], axis=1)\n",
    "\n",
    "feat_names = X_test.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_test = csr_matrix(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:35:04.302203Z",
     "start_time": "2020-06-27T12:29:28.640880Z"
    }
   },
   "outputs": [],
   "source": [
    "#load训练好的模型\n",
    "model = pk.load(open(\"Otto_LightGBM_goss_org_tfidf.pkl\", 'rb'))\n",
    "\n",
    "#输出每类的概率\n",
    "y_test_pred = model.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:42:15.416276Z",
     "start_time": "2020-06-27T12:42:15.412285Z"
    }
   },
   "source": [
    "### 生成提交结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:35:11.236429Z",
     "start_time": "2020-06-27T12:35:11.229455Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(144368, 9)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_pred.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-27T12:35:17.771097Z",
     "start_time": "2020-06-27T12:35:15.398339Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "out_df = pd.DataFrame(y_test_pred)\n",
    "\n",
    "columns = np.empty(9, dtype=object)\n",
    "for i in range(9):\n",
    "    columns[i] = 'Class_' + str(i+1)\n",
    "\n",
    "out_df.columns = columns\n",
    "\n",
    "out_df = pd.concat([test_id,out_df], axis = 1)\n",
    "out_df.to_csv(\"LightGBM_goss_org_tfidf.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提交kaggle网站得分如下：<br>\n",
    "![](http://qckw6brve.bkt.clouddn.com/2.jpg)<br>\n",
    "Private Leaderboard排名第715位<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
